Searching for a material platform to realize the control of chiral phonons relevant to the ferroics is practically significant for information application including novel phononic devices1,2. The newly-emerging van der Waals (vdWs) layered breathing kagome semiconductor Nb3X8 (X=Cl, Br, I) featuring a 2D network of corner-sharing triangles is theoretically proposed to possess emergent properties including ferro-electricity/magnetism, and ferrovalley, but lacks of correlative experimental evidences3-6. Here, we visualize the intriguing unconventional atomic arrangement with alternating vdWs gaps in Nb3I8, based on which we confirm the phonon chirality and realize the electric-field induced antiferroelectric-ferroelectric transition at room temperature. The phonon chirality is tied with the circular vibration of breathing lattices, and the origin of the ferroelectricity is from the local net dipole moment reversal due to the size switching of the alternate triangles involving the rearrangements of intralayer ferrielectric-like and interlayer antiferroelectric-like orders. The coexistence of phonon chirality and antiferroelectricity/ferroelectricity in Nb3I8 suggests a promising exploration of multiferroicity in breathing kagome lattices via the control of the phonon chirality, which should be of importance for designing the phonon-based information devices.