The initial ten acres of land for St. Mary's Cemetery was purchased in 1863 by Father Martin Dowling, pastor of Our Lady of Mercy Parish in Port Chester. The cemetery was a private burial ground for Catholics before the purchase. The earliest recorded burial is of John Miley in 1854, the year the parish was established. Permanent roadways were installed in 1927 and additional property was added in 1945 and 1947. To accommodate a growing need, beautiful communal mausoleums were constructed, beginning in 1971. Expansion and new construction took place in several phases through 2005. There are two soaring chapel-like crypt rooms with comfortable seating, flower stands and stained glass windows that honor Our Lady of Mercy, the Infant Jesus and the Resurrection. The main floor of the mausoleum features a bronze crucifix sculpted by Enrico Arrighini from the image of Christ on the Shroud of Turin. Among the many unique headstones in the cemetery, 500 were carved by Luigi Del Bianco, chief carver of the Mount Rushmore National Memorial in South Dakota.

As a result of the Making All Things New pastoral planning process, Our Lady of Mercy merged with three other parishes in Port Chester to become St. John Bosco Parish. In 2018, St. Mary's Cemetery was transferred to the Trustees of St. Patrick's Cathedral, the entity that has been responsible for cemeteries of the Archdiocese of New York since 1848.

To date, more than 25,000 Catholics and members of their families have been laid to rest in the 18.37 peaceful acres of St. Mary's Cemetery. They are interred in graves, private family and community mausoleum crypts and in cremation niches located within the community mausoleums.