Replying to What should I do?

So I just started building my summer setup two weekends ago, and was pretty excited to finally be able to ski again, especially at my own house. So while I am done with the tower and adding on the supports and such, my neighbor comes to see what it is. Hes some old guy that i've never really talked to much. He just asks what it is and i try to explain the best i can. He doesn't seem too interested and walks away. So his house is behind mine, and for the record you can barely see the setup from his house, and he has many trees blocking it in his backyard.

So i didn't take much from this conversation, and kept building. I finished soon after and could actually ski it. But the next week i got a letter in the mail from my town, saying they got a complaint and that i cannot have a "recreational facility" in my yard. This apparently violates the "zoning code". So now i have to set up a meeting with these people, and i really don't want to have to take down my setup due to some asshole neighbor.

What should I do?