Southampton is a ghost town in the off-season. Locals have a name for the day after Labor Day: “Tumbleweed Tuesday”. I don’t know if I’ve ever seen an actual tumbleweed on Tumbleweed Tuesday, but the name is fitting—there’s an eerie stillness in the air throughout the Hamptons. Traffic is lighter. Noise is quieter. Customers are fewer. At the end of another exhausting season, locals contemplate the death of summer with relief. Of course, “Summer” for us is less the actual summer season, and more the period between Memorial Day and Labor Day—when hordes of homeowners and vacationers flock to the Hamptons en masse and descend upon the region like seagulls, bringing wealth and parties and traffic and (most importantly) job opportunities. And then, on Tumbleweed Tuesday, most of them vanish. It’s funny, actually, having spoken to some of them—they sort of assume that Southampton itself vanishes with the season too. Friends who have visited and friends who own houses have asked me, “Wait—people actually live there year-round?” They do. People grow up and go to school and live their lives not far from rows of empty mansions, and I was one of them.

I was born in Southampton about a year after my mother arrived from Costa Rica with a cursory knowledge of English and absolutely no idea what “The Hamptons” meant to the average American. All those empty houses need cleaning, on and off-season, so that’s what she did for work until I was around ten. My biological father painted houses, and I think he may have done some landscaping as well. This is very common—a lot of immigrant workers are hired off-the-books for jobs of this sort. They generally rent houses or apartments, or even just rooms in other people’s houses. I remember the house I lived in until I was five—it was a small, old building that housed up to four families at one point. My dad used to rent the house from a landlord, and then rent individual rooms to several other families and migrant workers, so that from their rent he could pay the house’s. All of them would leave for work during the day, and on days my mother didn’t take me with her, I’d be left to haunt the rented property by myself. Until the age of five, this house—and the mansions my mother took me to—were the entirety of my world. We didn’t interact with anyone that wasn’t also Hispanic, and we mostly kept to ourselves anyhow. I didn’t know a word of English.

I suppose growing up in the Hamptons as a local is already unusual, but there’s something particularly spectral about growing up as the son of a housecleaner. For example, my mother used to let me swim at houses that had pools while she cleaned them, as long as she was sure the owners weren’t in the area. I taught myself to swim in a wealthy stranger’s pool. It was always like a holiday when my mom would tell my sister and me, “Get your bathing suit.” Even after we moved from that first, crowded home, we never got to rent a house with a pool. My sister and I dreamed growing up what it must be like to have a pool at your own house. All this is to say, I lived a large cultural distance away from the owners of the large, impressive houses my mother used to clean, even though we lived not far from them. I never knew what it was like to “summer” in the Hamptons. (Only now, in college, do I get to use “summer” as a verb in jest, when I go home for summer break.) But I did always have a view of their world—the world of summer vacationers, the world of obscenely wealthy mansion owners, of realtors and art collectors and the occasional celebrity—albeit at that cultural distance.

We also didn’t have a pet until much later on, but one house had an orange tabby cat I used to play with. His name was Dylan. Why the owner chose that name for a cat is beyond me, but the man had a stroke a couple years after my mother started working for him. She stopped cleaning the house, and we got to keep the cat. Other hand-me-downs came from the owners themselves. Sometimes it was old appliances. Often it was old clothes, from brands far beyond our budget. Most of my baby clothes were hand-me-downs from richer folk, my mother told me. One time, when she spotted the store where a pair of gifted overalls were from, she walked in and nearly started crying when she checked the first price tag.

My mom stopped working at these houses long ago, but the last of these gigs she had a few years back, taking care of an ailing Broadway composer in his old age. His name was Richard Adler—he composed the music to “Damn Yankees” and “The Pajama Game”. Ever heard “Whatever Lola Wants”? You likely have. He wrote it. His widow gave me his Brooks Brothers navy jacket after he passed. I had it tailored, and keep it with me at Princeton now. This is one of the remnants of this spectral, earlier life that still persists—you can catch me wearing a dead man’s jacket at any formal function.

But most of the remnants of this past existence are simply memories, each one grounded in a different house my mother cleaned. Each one had a different character, and each one contributed some sort of story to my childhood memories. Like I said, there was the house where I learned how to swim. There was another where the owner decorated the rooms by themes. One room had furs and large, black, wooden idols on the walls. This room is one of the first things I can remember being afraid of. There was another house—and my mother remembers this one—where faucets would open and doors would close on the second floor while no one was there. We’d close the faucets and go back downstairs and they’d open again. One time I crept quietly up the stairs to shut one, listening closely, and managed to make out a soft murmur coming from one of the walls. I tripped and the noise must have scared him away.

I have to say, though, none of this felt unfamiliar. Yes, the houses were far different from anything I ever lived in, and what I gleaned from the view into the lives of these homeowners was something I don’t think I’ll ever afford for myself. But I never felt alien inside them, I never felt like some intruder or impostor. It was always a very familiar feeling walking through these mansions. In a sense, I grew up in them as much as in my own house. How could you feel like an outsider in the place you grew up?

But I was still far from being an insider, in more ways than one. This is what makes it spectral. As a local, I didn’t take part in this summer homeowners’ world, and as the son of immigrants—with my entire family living thousands of kilometers away, growing up for the first half of my childhood culturally removed from general society—I did not feel fully “American”. I also did not feel fully Hispanic. It’s hard to put this into words—the point is that I grew up in Southampton in a sort of in-between state, in a kind of limbo, familiar yet foreign, seen and unseen. Call it what you may. I can identify with every “Things only Hamptons Kids Will Get” listicle, but I also get to recount stories even Hamptons kids don’t understand.

I presume I’m not the only one that grew up this way. I’ve never met any that have spoken about it, though. It’s a hard thing to express, even if you take the time to reflect on it. My sister, who grew up alongside me, hasn’t given it much thought. I’ve tried to tease out a couple stories from Latino friends of mine, to see what the Hamptons they grew up in were like, and they recount similar stories. One person’s childhood is grounded in the memory of their rented house, a house where the long-deceased original owner would often walk around the property. His apparition could be seen in the yard from time to time. Nobody minded, apparently, but why would you? Work for immigrants and unskilled workers is so scarce on the off-season, there are usually better things worth minding.

Now that I think about it, none of these phantoms seem to show up during the summer. Maybe it’s just too loud? Maybe the season’s too hard on ghosts. It usually is on me, and I find my part time job mostly bearable. It used to be worse for my mom when she cleaned houses and lost clients on the off-season. It’s still bad for those that depend on summer work to feed their families. I’ll put it to you, readers, to let me know. Those of you that have ever visited the Hamptons in the summer, have you encountered any ghosts? (Did you encounter the housecleaner’s son? I thought not.) Those of you “Hamptons kids”—friends from home reading this article, perhaps—have any murmurs echoed in your house from sources unexplained? Have you caught a glimpse of any silver shadow, of any spectral figure, standing around a corner? The ones at Camp Hero in Montauk don’t count, of course—those show up every month of the year, and everyone knows about those.

Life in Southampton has changed for my sister and me, since those days of living in a ratty rented shack. My sister and I now live in a nice house—it’s small, and far from a mansion or anything fancy, but it does have a pool. We’ve managed to find our own place in this little ghost town. Even though I don’t think I’ll ever afford to live here if not with my mother and stepfather, a little piece of my heart lingers on here. I’m sure I’ll visit often, though. Make an apparition every now and then, perhaps. Maybe someday, after years of not being able to own a house of my own here, when I’ve passed away, I’ll be spotted smiling somewhere by the bay, back at the old haunt—in my twenties of course, that’s the kind of ghost I’ll be. I’ll be one of the ones that don’t mind the summer. After all, though it is hectic, it is the nicest time of year. Until that day I’ll live content simply remembering the place I grew up in, conjuring up the spirits of old friends and landlords, of lavish houses and haunted hand-me-downs, putting to use the one thing Southampton has taught me to do well and seeking new places to haunt.