The action of the novel is divided between Paris and Issoudun. Agathe Rouget, who was born in Issoudun, is sent to be raised by her maternal relatives, the Descoings in Paris by her father Doctor Rouget. He suspects (wrongly) that he is not her true father. There she marries a man named Bridau, and they have two sons, Phillipe, and Joseph. Monsieur Bridau dies relatively young, Phillipe, who is the eldest and his mother's favourite, becomes a soldier in Napoleon's armies, and Joseph becomes an artist. Phillipe, the elder son is shown to be a courageous soldier, but is also a heavy drinker and gambler. He resigns from the army after the Bourbon Restoration out of loyalty to Napoleon. Joseph is a dedicated artist, and the more loyal son, but his mother does not understand his artistic vocation.
After leaving the army Phillipe took part in the failed Champ d'Asile settlement in Texas. On returning to France he is unemployed, and lives with his mother and Madame Descoings, and becomes a financial drain on them, especially due to his hard drinking and gambling lifestyle. Phillipe becomes estranged from his mother and brother after stealing money from Madame Descoings. Phillipe is soon afterwards arrested for his involvement in an anti-government conspiracy.
Meanwhile in Issoudun, Agathe's elder brother Jean-Jacques takes in an ex-soldier named Max Gilet as a boarder. Max is suspected of being his illegitimate half brother. Max and Jean-Jacques' servant Flore Brazier work together to control Jean-Jacques. Max leads a group of young men who call themselves "The Knights of Idleness" who frequently play practical jokes around the town. Two of these are against a Spanish immigrant named Fario, destroying his cart and his grain, and therefore ruining his business.
It is now that Joseph and his mother travel to Issoudun to try to persuade Jean-Jacques to give Agathe money to help cover Phillipe's legal costs. They stay with their friends the Hochons. Jean-Jacques and Max only give them some old paintings, but only Joseph recognises their value. Joseph tells of his luck to the Hochons, not realising that their grandsons are friends of Max. Afterwards when Max discovers the value of the paintings he coerces Joseph into returning them. Then one night whilst out walking Fario stabs Max. As Max is recovering he decides to blame Joseph for the stabbing. Joseph is arrested, but later cleared and released, and he and his mother return to Paris.
In the meantime, Phillipe has been convicted for his plotting. However, he cooperates with authorities and gets a light sentence of five years Police supervision in Autun. Phillipe gets his lawyer to change the location to Issoudun in order to claim his mother's inheritance for himself. He challenges Max to a duel with swords, and kills him in the duel. He then takes control of Jean-Jacques and his household, forcing Flore to become Jean-Jacques' wife.
Phillipe marries Flore after the death of Jean-Jacques. Flore too soon dies. The book hints that both of these deaths are arranged by Phillipe but is not explicit about the means. Through his connections, Phillipe has now obtained the title Comte de Brambourg. Phillipe later marries a rich man's daughter. An attempt by Joseph to reconcile Phillipe and their mother before her death fails. Phillipe's fortunes take a turn for the worse after some unsuccessful speculation, and he rejoins the army to take part in the war in Algeria where he is killed in action, so that in the end Joseph, now a successful artist, inherits the family fortune.