William Kellie Smith was from a village in Scotland known as Kellas. In 1890, at the age of 20, he arrived in the then undeveloped Malaya. Here, he met an estate owner called Alma Baker, who had won concessions from the state government to clear 360 hectares of forests in Perak. With the substantial profits made from his business venture with Alma Baker, Smith started planting rubber trees and dabbled in the tin mining industry. In time, he became the owner of Kinta Kellas Estate and the Kinta Kellas Tin Dredging Company.

Now with his fortune made, he returned home to marry his Scottish sweetheart, Agnes, and brought her over to Malaysia in 1903. The following year, the couple was blessed with a daughter whom they named Helen. For many years after that, Agnes tried to conceive, but to no avail. William Smith desperately wanted a son and heir to take over his empire in the Malay Isles. After many years, Agnes finally gave birth to a son, Anthony, in 1915. The birth of his child was the start of even greater success for William Smith. To celebrate Anthony's birth, William Smith decided to expand on his mansion. Smith started planning for a huge castle which he planned to call Kellas House, after his hometown in Scotland.

Because of his fascination with the Hindu religion and the Indian culture, Smith's plan was for this house to share similar architecture to those of Madras, with all its bricks and tiles imported from India. He even employed a big group of Indian labourers to build his dream house, to keep the Kellas House authentically Indian. The mansion is accessible from the main road through a bridge running across a stream.

But it was not only the cost of importing material and labourers from abroad that made the house so fascinating to locals and travellers alike. Among the many amazing things about Kellie's Castle are an elevator (it was the first in Malaya) which connects right up to the top floor, and the existence of two tunnels that run under the river nearby. One of these tunnels connects to the Hindu temple some distance away from the main house. On the second floor, Smith planned to build an indoor tennis court ?an ambitious project even by today's standards. On the highest floor, there is a rooftop courtyard for parties. This castle was to be the hub for entertaining wealthy colonial planters who had settled in Malaya. His house was so unique that it was even mentioned in the London Financier newspaper on 15 September 1911.

Unfortunately for Smith, tragedies struck soon after the construction of the Kellas House began. A virulent strain of the Spanish flu spread from Europe to Asia soon after World War I ended in Europe, killing many of the workers in the Kellas Estate. Another seventy workers constructing Smith's dream castle also became victims of the flu. Smith, who had already spent a fortune on his house, lost a lot of money because of this.

In the end, Kellas House, later known as Kellie's Castle or even Kellie's Folly to some, was never completed. William Kellie Smith himself died of pneumonia during a short trip to Portugal in 1926. His heartbroken wife decided to pack up and return home to Scotland selling the estate and Kellie's Castle to a British company called Harrisons and Crosfield.