This paper proposes an optimization approach of a grid-connected photovoltaic and wind hybrid energy system including energy storage considering voltage fluctuation in the electricity grid. A techno-economic analysis is carried out in order to minimize the size of hybrid system by considering the benefit-cost. Lithium-ion battery type is used for both managing the electricity selling to the grid and reducing voltage fluctuation. A new technique is developed to limit the voltage perturbation caused by the solar irradiance and the wind speed through determining the state-of-charge of battery for every hour of a day. Improved particle swarm optimization (PSO) methods, referred to as FCVACPSO which combines Fast Convergence Particle Swarm Optimization (FCPSO) method and Variable Acceleration Coefficient Based Particle Swarm Optimization (VACPSO) method are used to solve the optimization problem. A comparative study has been performed between standard PSO method and PSO based methods to extract the best size with the benefit cost. A sensitivity analysis has been studied for different kinds and costs of batteries, by considering variable and constant state-of-charge of battery. The simulations, performed under Matlab environment, yield good results using the FC-VACPSO method regarding the convergence and the benefit cost of the hybrid system.