[User] Please transcribe this text

[Assistant] This was Miso Eltec, not his eldest son Furlan, and such dissuasion wouldn't convince him.


[User] Please transcribe this text

[Assistant] Where are you going in this armor? The old knight asked after short silence. To win honor and status. Misa wore a cuirass made of stainless steel.


[User] Please transcribe this text

[Assistant] Morning light, the top night in the western region, only one honor for himself. While I fight for my whole family.


[User] Please transcribe this text

[Assistant] Father, when this is finished, you'll know that I'm as worthy of respect as my brother.


[User] Please transcribe this text

[Assistant] If Timothy vanquishes Prince Roland, your words may come to pass.


[User] Please transcribe this text

[Assistant] However, the army of the new king is still far away while Prince Roland's army is very near, the old knight thought. Don't you remember how Duke Ryan collapsed? Eltech said in a low voice. Things are different this time.


[User] Please transcribe this text

[Assistant] The king has shipped a number of snow powder weapons, the same ones used by Prince Roland. He'll provide better ones to Med.


[User] Please transcribe this text

[Assistant] Without the possession of superior weapons, can Roland seize Longsong's stronghold again?


[User] Please transcribe this text

[Assistant] Miso clipped his sword to his waist and nodded to his father. The battle is coming soon. Please wait here for good news. Looking at his back, LTAC threw his crutch to the ground.


[User] Please transcribe this text

[Assistant] My lord, why didn't you stop him? The steward came up to hold the trembling knight.


[User] Please transcribe this text

[Assistant] A noble attending the rebellion meeting hadn't shown up at the agreed place. What did this mean for the four families?


[User] Please transcribe this text

[Assistant] The crime of escape or defection could lead the whole LTCH family to death.


[User] Please transcribe this text

[Assistant] Look on the bright side. If the four families win, the stew are comforted.


[User] Please transcribe this text

[Assistant] After all, the months of the demons are long and it'll take two or even three months for the snow to melt.


[User] Please transcribe this text

[Assistant] When the army of Prince Roland arrives, the king's army may have already besieged the western region. If it were the past, he would view it in the same way.


[User] Please transcribe this text

[Assistant] However, after having visited Border Town once, the night had a totally different view of Prince Rowland.


[User] Please transcribe this text

[Assistant] Not to mention his eldest son, Furlan Eltech, and his family's benefactor, Lady Agatha, were all living in Border Town.


[User] Please transcribe this text

[Assistant] He turned to his steward and told him in a stern voice, Go to border town immediately and deliver this news to his highness.


[User] Please transcribe this text

[Assistant] If the city gate is closed, go to the docks in the outskirts and find some ferrymen or fishermen. It doesn't matter how you do it, just do it quickly.


[User] Please transcribe this text

[Assistant] Before the housekeeper left the LTEC territory, Roland received a letter from the 2nd Army of Stronghold delivered by a bird.


[User] Please transcribe this text

[Assistant] An hour later, Petrov's call for help also reached the town's castle.


[User] Please transcribe this text

[Assistant] Since the carrier pigeon could only carry limited information, Roland had to combine the two messages and discovered a piece of astonishing news. The four families of the western region had rebelled.


[User] Please transcribe this text

[Assistant] Who on earth gave them the confidence to plot treason under my nose?


[User] Please transcribe this text

[Assistant] Roland immediately summoned Carter in iron axe and showed them the notes. Your highness is this. A message from stronghold, Roland said angrily.


[User] Please transcribe this text

[Assistant] The winter was drawing to an end, and the new year was in three days.


[User] Please transcribe this text

[Assistant] Not only were there various matters concerning the city construction, but also the installation of the equipment for the three supplies project required as instruction.


[User] Please transcribe this text

[Assistant] Those nobles chose to trouble me at the wrong time. They must be looking forward to dying.


[User] Please transcribe this text

[Assistant] According to the secret letter, the inner city nobles led a bunch of squires to attack the city guards.


[User] Please transcribe this text

[Assistant] By the time the sentries on the city wall realized something was wrong, the switch for the northern drawbridge had fallen into the control of the enemies.


[User] Please transcribe this text

[Assistant] The sentries could do nothing but light the beacon fire and rely on the city wall to hold their positions.


[User] Please transcribe this text

[Assistant] When the other members of the 2nd Army saw the beacon signal, they followed the emergency response protocol by dividing the camp's 10 teams of about 50 soldiers in total into two platoons.


[User] Please transcribe this text

[Assistant] One platoon charged toward the north gate, while the other went to the stronghold castle.


[User] Please transcribe this text

[Assistant] Petrov's secret letter was sent after the second platoon confronted the enemies at the castle and realized they were fighting the local nobilities.


[User] Please transcribe this text

[Assistant] Petroff-Hull mentioned that the attackers sieging the castle were carrying the Four Families' banners, which means this rebellion was planned. Carter frowned after reading the notes.


[User] Please transcribe this text

[Assistant] Since the four families are acting together, there must be a leader among them.


[User] Please transcribe this text

[Assistant] Indeed, the four families, or five families including those under Duke Ryan, were great nobles on equal footing.


[User] Please transcribe this text

[Assistant] If they were to set aside their prejudices and cooperate, it was most likely because a bigger force was behind them.


[User] Please transcribe this text

[Assistant] Otherwise, with the Earls looking down upon each other, it would take over a decade for them to decide on a leader.


[User] Please transcribe this text

[Assistant] As a member of the Iron Sand people of the Majin clan, he was far less sensitive to the political situation of the Kingdom of Greycastle than Carter was.


[User] Please transcribe this text

[Assistant] I wouldn't be surprised if it's the church. Roland twitched his mouth and said,


[User] Please transcribe this text

[Assistant] Judging from their actions at Fallen Dragon Ridge, it's only a matter of time before they attack the western region.


[User] Please transcribe this text

[Assistant] The problem is, in the second letter Petrov mentioned the enemies might have firearms. Does it mean they have learned the formula for gunpowder?


[User] Please transcribe this text

[Assistant] Snow powder isn't unknown to the church in Timothy, but they have a long way to go before they can use it smoothly. The prince knocked on the table lightly.


[User] Please transcribe this text

[Assistant] What puzzles me is that with Border Town as the nearest town to Stronghold, the help they get must be from somewhere further.


[User] Please transcribe this text

[Assistant] So even with the support of an unknown force, how did they gain enough confidence to confront the First Army?


[User] Please transcribe this text

[Assistant] Probably because we're in the months of the demons, Iron Axe replied after contemplating for a while.


[User] Please transcribe this text

[Assistant] Although last time your highness defeated the dukes knights using thunder power, our troops didn't show extraordinary skills in battle.


[User] Please transcribe this text

[Assistant] Now with the snow blocking the roads and demonic beasts attacking the borders, it seems like perfect timing for a successful rebellion, and it'll be at least two months before you have the resources to attack Longsong Stronghold.


[User] Please transcribe this text

[Assistant] And I'll be hindered by the lofty city walls and powerful flintlocks. Yes, iron axe couldn't help but grin.


[User] Please transcribe this text

[Assistant] They have all the weapons that you have, plus the insurmountable city wall and reinforcement of an unknown force. So they have a very good chance of beating you at the foot of the city wall. That's probably what they thought. Yet they're fully unaware of the power of Border Town. Roland stood up. Iron Axe. At your command, your highness. Iron Axe instantly stood at attention. The First Army must immediately prepare for an expedition.


[User] Please transcribe this text

[Assistant] We'll set off tomorrow morning, Roland said decisively. Choose 500 soldiers. Carry six field artilleries. You are in charge of making the specific arrangements.


[User] Please transcribe this text

[Assistant] The scale of the First Army had quadrupled its original 500 soldiers to 2,200.


[User] Please transcribe this text

[Assistant] Revolving rifles were upgraded, and their firepower was drastically increased.


[User] Please transcribe this text

[Assistant] Besides, the efficiency of the 1st Army's executing instructions was greatly enhanced due to the improvement of the soldier's educational level.


[User] Please transcribe this text

[Assistant] Upon command, personnel would deploy ammunition and rations based on the number of soldiers on the operation and the number of days.


[User] Please transcribe this text

[Assistant] The logistics work for 500 people could be done in almost one day.


[User] Please transcribe this text

[Assistant] Compared with the Knights or the Mercenaries, who needed weeks to prepare, Roland's army was functioning at an incredible speed.


[User] Please transcribe this text

[Assistant] The army headed towards the inner city of Stronghold on 11 paddle steamers.


[User] Please transcribe this text

[Assistant] After entering mass production, the production of a boat with a reinforced concrete hull took only five days, which was even a deliberately lowered speed, because its number was limited by the available crew and steam engines.


[User] Please transcribe this text

[Assistant] At your command, your highness, the chief knight said and raised his head high. You stay in border town. The corners of the knight's mouth instantly fell down. Wha, what? No, your highness, how could you leave me behind again? We're still in the months of the demons.


[User] Please transcribe this text

[Assistant] Demonic beasts pose a greater threat than the nobility, Roland said unwaveringly. Guard the town well. After the two of them retreated, Roland sighed. This time the witches need to go with me.


[User] Please transcribe this text

[Assistant] I'll follow you, no matter where you go. Nightingale said with a smile.


[User] Please transcribe this text

[Assistant] Considering their opponent could be the church, Roland knew that only the witches could effectively counter the pure witches' attacks without God's stones of retaliation.


[User] Please transcribe this text

[Assistant] After debating his options, Roland decided on the witches who would accompany him. Lightning, Maggie, Sylvie, Nama, Lily, and Nightingale.


[User] Please transcribe this text

[Assistant] If I add the three combat witches from Sleeping Island to the list, it'll be very hard for the pure witches to defeat us.


[User] Please transcribe this text

[Assistant] Roland stood at the window and looked at the snowy mountains and plains.


[User] Please transcribe this text

[Assistant] If I view the impassable mountain range as the city wall of stronghold, then the barbarian land in the north is also the domain of the four families, a territory beyond the reach of my power.


[User] Please transcribe this text

[Assistant] Elk, wild rose, maple leaf, and wolf families have operated separately on their own lands for over a decade, forming their unique laws and customs. So even Ryan, the overlord of the western region, seldom interfered with their businesses, mused Roland.


[User] Please transcribe this text

[Assistant] The prince had intended to slowly divide and annex these aristocratic territories after building the new city.


[User] Please transcribe this text

[Assistant] But the rebellion gave him an early opportunity, one to swallow the vast land and population of Stronghold in one gulp. He didn't intend to let go of the nobles this time.


[User] Please transcribe this text

[Assistant] Cacuzum arrived at the dock as the first rays of sunlight started to peep through the clouds.


[User] Please transcribe this text

[Assistant] Unlike the usual, the dock was packed with silent soldiers.


[User] Please transcribe this text

[Assistant] They stood erect, with their sacks and long barreled guns on their backs, looking like a thick forest in a storm.


[User] Please transcribe this text

[Assistant] Although the dock was crowded, everyone was moving in an organized manner.


[User] Please transcribe this text

[Assistant] Watching the soldiers board the paddle steamer one by one, Capcusum was filled with an indescribable sense of power and strength.


[User] Please transcribe this text

[Assistant] He swallowed hard while thinking, these are the soldiers trained by his highness. How incredible.


[User] Please transcribe this text

[Assistant] The old man had traveled from Sea Windshire to the Port of Clearwater as a youth.


[User] Please transcribe this text

[Assistant] This distance was over half the kingdom of Grey Castle, and he had also led the commercial fleet to the fjords and nearby islands.


[User] Please transcribe this text

[Assistant] So, he had personally witnessed the arrogant demeanor of the Armored Knights, as well as the fierce demeanor of the barbarians who killed beasts barehanded.


[User] Please transcribe this text

[Assistant] To him, these soldiers were undeniable fighters that could exercise extreme power.


[User] Please transcribe this text

[Assistant] However, he had not expected to feel that power once again, but while standing with a group of ordinary people he could feel it and it was stronger than ever. No doubt these are just ordinary people. He thought,


[User] Please transcribe this text

[Assistant] It had been about four months since Cacuzum arrived at Border Town, and each day he understood the town a little better.


[User] Please transcribe this text

[Assistant] He knew that the First Army consisted mostly of locals, and that many of them were miners, hunters, furnace laborers, and masons before they joined the Army.


[User] Please transcribe this text

[Assistant] Because of this, they had never received any professional combat training.


[User] Please transcribe this text

[Assistant] Nevertheless, in just a few months, these people had become as brave and disciplined as any knight. What magic did his highness use on them? Are you really going? Cacusum heard Vader whisper from behind him.


[User] Please transcribe this text

[Assistant] He could tell from his lowered voice that he too was silenced by the presence of the army. Why did I apply for the captain position if I wasn't going? Cacusum answered while taking a deep breath. But they are off to fight. Vader continued. They all offer their services to his highness. The old man corrected. Without turning his head he added, and so do I.


[User] Please transcribe this text

[Assistant] Following the momentary silence, Vader then implored, Stay alive. Cacusum waved in response.


[User] Please transcribe this text

[Assistant] Cacuzum boarded the sixth paddle steamer as it arrived with the swirling snow.


[User] Please transcribe this text

[Assistant] According to the tradition, a captain can name their own ship, and even though this boat belonged to his highness, he was still allowed to decide its name. Yet, he still had not made up his mind.


[User] Please transcribe this text

[Assistant] This was the second time he had assumed the position of captain since his retirement 10 years ago.


[User] Please transcribe this text

[Assistant] So, he wished to come up with a name dedicated to his memory. Captain, there you are.


[User] Please transcribe this text

[Assistant] As soon as Kakuzim stepped into the cabin at the helm, the first mate came up to greet him.


[User] Please transcribe this text

[Assistant] We are now preheating the boiler and I promise it will soon be ready to go. The young man was called Pike and he was from the southern territory. He had a few years experience fishing on the sea.


[User] Please transcribe this text

[Assistant] If you don't know how to respect your captain, I'll be happy to teach you through a full day of deck cleaning. Yes, captain.


[User] Please transcribe this text

[Assistant] The young man shouted, erecting himself immediately, of course I do. That's better, Cacuseum commented. While stroking his beard he gave pike orders.


[User] Please transcribe this text

[Assistant] Tell the boiler house to stoke up the fire but not to shut the damn steam valve. I don't want to rear end the boat in front of us. Yes, gotcha.


[User] Please transcribe this text

[Assistant] Without finishing the sentence, Pike had slipped back into his previous playful manner.


[User] Please transcribe this text

[Assistant] He quickly winked at the old man and dashed out of the cabin.


[User] Please transcribe this text

[Assistant] That rascal, Cacusa mumbled while shaking his head smiling.


[User] Please transcribe this text

[Assistant] He was more at ease after the interaction, as if he had slipped back to the good days of when he roamed the sea.


[User] Please transcribe this text

[Assistant] The old man went back to the wheel, where he gently stroked its wooden handle.


[User] Please transcribe this text

[Assistant] Slowly, he began to remember the operating procedures for a paddle steamer.


[User] Please transcribe this text

[Assistant] It didn't have a mast or a cabin below deck. Instead, it had two chambers.


[User] Please transcribe this text

[Assistant] The first chamber is located at the helm and is called the wheelhouse.


[User] Please transcribe this text

[Assistant] This room has two large windows from which the captain can see the route and navigate clearly.


[User] Please transcribe this text

[Assistant] The second chamber is found in the middle and it contains the boiler that powers the boat.


[User] Please transcribe this text

[Assistant] Behind the wheelhouse, there was a bare deck and this space was often filled by miners during training.


[User] Please transcribe this text

[Assistant] During this time of training, they would often travel west along the Redwater River to drop the miners off at the edge of the misty forest, where they could find coal.


[User] Please transcribe this text

[Assistant] Coal lasted longer than wood and was the preferred fuel for Sea Windshire.


[User] Please transcribe this text

[Assistant] Now, there was a makeshift shed constructed from cloth, apparently set up in preparation for the boarded soldiers.


[User] Please transcribe this text

[Assistant] Although Cacuseum was not familiar with the stone boat, he soon realized it was not hard to operate and even simpler than a sailboat in many ways.


[User] Please transcribe this text

[Assistant] First of all, this boat didn't require wind and sails for direction and power. Overall, it took less manpower to run.


[User] Please transcribe this text

[Assistant] Furthermore, it would not be hard to teach a villager how to run a stove, whereas it would take at least six months for them to master sailing.


[User] Please transcribe this text

[Assistant] The stone boat could run by itself for a long time as long as the engine was on and the steam valve was shut.


[User] Please transcribe this text

[Assistant] Just then, the tranquility of border towns early morning was broken by a dull, steam whistle from the front of the line. The first boat was sailing off.


[User] Please transcribe this text

[Assistant] Captain, the water in the boiler is ready. Reported Pike who scurried back to the wheelhouse.


[User] Please transcribe this text

[Assistant] Ring the bell to tell Big Pad and Grizzly that it is time to shut the valve and speed up. It's time to move. Cacuzum solemnly gave the instructions.


[User] Please transcribe this text

[Assistant] Pike pulled on the long iron string attached to the wall, which would ring the bell in the boiler house and deliver the captain's command.


[User] Please transcribe this text

[Assistant] In response, the boat shook violently and the wooden tires on either side started to move slowly.


[User] Please transcribe this text

[Assistant] Cacuse him held tight to the wheel at the helm and looked straight ahead.


[User] Please transcribe this text

[Assistant] When Vader asked him why he wanted to fight, he actually did not tell him the truth.


[User] Please transcribe this text

[Assistant] Saying it was to serve his highness was only a small part of the reason. The truth was he simply enjoyed being a captain.


[User] Please transcribe this text

[Assistant] It didn't matter if it was a sailboat or a paddle steamer, Cacusum truly loved the feel of the wheel in his hands as he navigated the bow through the thrusting waves. This is the life he really wants. Full sail. No, continue to shovel coal. The old man turned to face starboard while hollering. Hold on guys. We're setting out. If you ferry me to border town the Elks will pay you well.


[User] Please transcribe this text

[Assistant] The steward asked while blocking the troubled boatman's cabin door.


[User] Please transcribe this text

[Assistant] Your Excellency, I'm happy to be at your service, but I just can't. The Boatman Stammered, L.


[User] Please transcribe this text

[Assistant] Look, there isn't even a shed above my boat to protect you from the snow.


[User] Please transcribe this text

[Assistant] It wouldn't be a problem to ferry you across the river, but to border town, that would take several days. Just consider this freezing weather. Where would we sleep, the boatman implored.


[User] Please transcribe this text

[Assistant] We would become frozen like popsicles in just one night on the boat, wouldn't we?


[User] Please transcribe this text

[Assistant] Are there any other boatmen nearby who can ferry me to border town? The steward persisted.


[User] Please transcribe this text

[Assistant] You should go to Stronghold to find a boat you can spend the night on.


[User] Please transcribe this text

[Assistant] If I could have entered Longsong Stronghold, why did I come here looking for a boatman that fished and ferried for a living? The steward thought to himself as the boatman shut the door. The steward kicked at the snow.


[User] Please transcribe this text

[Assistant] Ever since the Four Families started to attack Stronghold, all the city gates had been closed.


[User] Please transcribe this text

[Assistant] He spent a long time taking this detour and now found that all of his efforts were in vain. It will be getting dark soon. How am I going to complete my lord's task?


[User] Please transcribe this text

[Assistant] The steward sullenly looked at the Redwater River, but he was soon stunned. Good heavens, what's that?


[User] Please transcribe this text

[Assistant] He rubbed his eyes in disbelief, making sure what he was seeing was not an illusion.


[User] Please transcribe this text

[Assistant] A huge fleet was coming down the river and the boats were unlike any he had seen before.


[User] Please transcribe this text

[Assistant] Through the dense misty snow, the vessels roared and whistled towards him at full speed.


[User] Please transcribe this text

[Assistant] There was not any sail on the gray boats, and yet they were still proceeding against the wind, their bows cutting the water and cleaving through the waves.


[User] Please transcribe this text

[Assistant] On the first boat, he could see a flag flapping in the wind with the embroidered emblem of a tower and a gun.


[User] Please transcribe this text

[Assistant] The steward held his breath in realization. This is the fleet of Prince Roland Wimbledon.


[User] Please transcribe this text

[Assistant] Long Song Stronghold was bisected by a branch of the Redwater River, to the west laid the residential area, and to the east laid vast farming lands.


[User] Please transcribe this text

[Assistant] Unlike the other big cities that were set in the suburbs of farmlands, the towering city wall of the stronghold encircled both the residential area and a portion of the farmlands.


[User] Please transcribe this text

[Assistant] It was built to defend against human beings and not demonic beasts.


[User] Please transcribe this text

[Assistant] However, even if the city was under siege, Stronghold would still flourish and be able to supply its people.


[User] Please transcribe this text

[Assistant] As such, the outer wall of Longsong Stronghold was split where the river divided the two sections of the city.


[User] Please transcribe this text

[Assistant] Between the divide and the walls hung several massive chains, which were the width of a man's arm.


[User] Please transcribe this text

[Assistant] If an enemy decided to attack from the river, the hemp cords holding the heavy chains would be cut and send them crashing into the water to stop the enemy below.


[User] Please transcribe this text

[Assistant] Fortunately, the inner land of the western region had rarely been attacked by any fleets.


[User] Please transcribe this text

[Assistant] So, these cables had almost never come into use, and would not this time either.


[User] Please transcribe this text

[Assistant] Out on the water, Roland's fleet pushed through the storm.


[User] Please transcribe this text

[Assistant] Around twilight, they were able to see the silhouette of the fortress in the distance.


[User] Please transcribe this text

[Assistant] Strands of the beacon's fire could be seen flashing above the city, slashing through the inkiness of the ever-gradually darkening sky.


[User] Please transcribe this text

[Assistant] The enemies have already reached the Lord's Castle and there are some militias fighting with the guards at the north gate of the city wall. It looks like they can't hold up any longer.


[User] Please transcribe this text

[Assistant] Lightning, who had carefully investigated the situation in the city, continued giving her detailed report to the Prince. There are around 200 people attacking the castle. The first round of attacks failed.


[User] Please transcribe this text

[Assistant] His family was killed, lightning answered, her mouth twitching.


[User] Please transcribe this text

[Assistant] His Excellency Petrov warned his father and asked him to get to the castle.


[User] Please transcribe this text

[Assistant] However, his other family members were taken as hostages, including the Countess.


[User] Please transcribe this text

[Assistant] They were being held by the four families in exchange for surrender and subsequently they were executed one by one.


[User] Please transcribe this text

[Assistant] The little girl paused for a moment before continuing. I took an opportunity to fly into the castle to tell him that you would be coming soon and he had only one request. Roland nodded and asked, What is it? Blood for blood.


[User] Please transcribe this text

[Assistant] There was an unwritten rule among nobles where people with bestowed titles were exempt from being murdered in times of battle.


[User] Please transcribe this text

[Assistant] In the case of the family of Earl Honeysuckle, most of his loved ones had been honored with a knighthood. However, the four families had betrayed that rule.


[User] Please transcribe this text

[Assistant] Not only did they hate the Honeysuckle family because of the benefits of Roland's support, but they were also being driven by a new political power.


[User] Please transcribe this text

[Assistant] In light of this, Roland was almost certain the power behind them was the new king, Timothy Wimbledon.


[User] Please transcribe this text

[Assistant] He now realized there was no room for mercy and that he needed to crush them.


[User] Please transcribe this text

[Assistant] Supporting the king and executing rebels was very different from helping the church slaughter nobles.


[User] Please transcribe this text

[Assistant] He raised the flintlock numbly, aiming it at the dark hole below.


[User] Please transcribe this text

[Assistant] This was the only path to climb up the city wall from within.


[User] Please transcribe this text

[Assistant] There had been several rounds of attacks since the enemies captured the city gate, but they were still not able to seize the northern part of the city wall.


[User] Please transcribe this text

[Assistant] The battle started yesterday at noon and had yet to cease.


[User] Please transcribe this text

[Assistant] He hadn't expected to be standing in the cold brisk wind for so long.


[User] Please transcribe this text

[Assistant] After watching his supporting comrades fall one by one into pools of blood, all he could focus on was fighting. Are they coming again?


[User] Please transcribe this text

[Assistant] Seeing a long-barreled gun pop up next to him, 8 tilted his head to the side and found that it was his unit leader, Bronze Hill.


[User] Please transcribe this text

[Assistant] I could hear them, eight replied feebly, but I have no idea how many there are in the darkness.


[User] Please transcribe this text

[Assistant] They don't know how many of us are left either. Bronze Hill comforted him. So, how many are left?


[User] Please transcribe this text

[Assistant] And we probably won't see the sun tomorrow, 8 said bitterly.


[User] Please transcribe this text

[Assistant] After the initial feelings of fear and panic, he then became numb.


[User] Please transcribe this text

[Assistant] It was this experience that helped him finally understand what his instructor had meant when he said, Only a real battle will quickly make you a man.


[User] Please transcribe this text

[Assistant] However, this did not make the current situation any better.


[User] Please transcribe this text

[Assistant] Out of the three units, there were only five people left.


[User] Please transcribe this text

[Assistant] Even if the enemies didn't make it up the wall, they would likely die from starvation or exposure to cold. Can we still surrender? Surrender? Bronzehill snorted. Didn't you hear what they were yelling? Kill the rebels. There's nothing but death ahead. We'd better take down as many as possible while we can. Eight knew the unit leader was right. Many captured soldiers from the second army were beheaded.


[User] Please transcribe this text

[Assistant] They were not nobles, and nobody would pay a ransom. But, he just hated the idea of dying up here.


[User] Please transcribe this text

[Assistant] The instructor had told them about how a happy life in Border Town was and promised that once the months of the demons had passed, soldiers who did well in the Second Army would be transferred to the First Army and receive free housing there.


[User] Please transcribe this text

[Assistant] There was plenty of food in Border Town, as well as running water and a heating system that could warm up a whole room without ignition.


[User] Please transcribe this text

[Assistant] In the muzzle flash, 8 could see the figure of the enemy revealed in the darkness.


[User] Please transcribe this text

[Assistant] Six or seven people were creeping slowly up along the wall, holding up their shields. He even caught a glimpse of the terror in their eyes.


[User] Please transcribe this text

[Assistant] The unit leader missed his target. Fortunately, he was already pointing the gun in the right direction. He aimed at the fleeting figure and fired again.


[User] Please transcribe this text

[Assistant] There was the muffled sound of a bullet hitting a wooden shield, and then a shriek, followed by the sound of a heavy object rolling down the stairs.


[User] Please transcribe this text

[Assistant] Realizing that they had been exposed, the rest of the enemies rushed to the opening recklessly.


[User] Please transcribe this text

[Assistant] Bronzehill was struggling to reload his gun, but his frozen hands hindered the process, slowing him greatly.


[User] Please transcribe this text

[Assistant] Eight jumped back from the open pit and groped frantically for his gunpowder sack.


[User] Please transcribe this text

[Assistant] His heart sank when he found it empty and he realized that he had already used it all.


[User] Please transcribe this text

[Assistant] According to his training, now is the time to use the bayonets.


[User] Please transcribe this text

[Assistant] He pulled out the bayonet and struggled to attach the knife to the front of his gun.


[User] Please transcribe this text

[Assistant] Clenching his teeth in frustration, he finally got it to attach and lifted it just in time as the enemy breached the hole and rushed at him.


[User] Please transcribe this text

[Assistant] Instantly firing, Bronzehill took the first man down, but there was a second close on his heels and this one managed to slide a sword into Bronzehill's chest. Eight was stunned and his training took over.


[User] Please transcribe this text

[Assistant] He mechanically thrust his bayonet in futility, only striking the enemy's shield.


[User] Please transcribe this text

[Assistant] The enemy easily kicked him to the ground disarming him completely. Am I really going to die here?


[User] Please transcribe this text

[Assistant] Looking up at the night sky in despair, he noticed a dark figure above him.


[User] Please transcribe this text

[Assistant] He normally wouldn't notice something so dark against the inky sky that night, but the shadow was close and rapidly getting closer still. It plummeted towards him like a collapsed curtain made of night.


[User] Please transcribe this text

[Assistant] Eight's eyes bulged as the shadow became more defined, and he realized it was a giant beast. This was a beast only found in nightmares.


[User] Please transcribe this text

[Assistant] Its huge crimson mouth was gaping, head larger than a bull's, and giant wings that almost covered the entire length of the city wall.


[User] Please transcribe this text

[Assistant] It released a thunder scrawl that was loud enough to wake the dead.


[User] Please transcribe this text

[Assistant] He helplessly stared at the beast, who now also turned to look at him. He could clearly see mucus dripping from its beige fangs. Relax. I'm not going to eat you.


[User] Please transcribe this text

[Assistant] The beast suddenly spoke in human language with a muddy voice, which almost horrified him enough to shriek.


[User] Please transcribe this text

[Assistant] She walked towards Bronze Hill, turned him over and checked him out. She said to the Beast, he's alive. Maggie, send in to Nana. Ow. The Beast seemed to understand what she said. It grabbed Bronze Hill with its claws and flew up again.


[User] Please transcribe this text

[Assistant] The flapping of its wings stir up the air, producing strong winds, which made Eight unable to open his eyes.


[User] Please transcribe this text

[Assistant] When the wind and snow finally stopped, he squinted at the city wall but saw nothing at all, as if everything that had happened was just a dream. No, it's not a dream. That girl is still here, he thought. He vaguely saw the girl's figure in the darkness.


[User] Please transcribe this text

[Assistant] She squatted down, dug out a long sword from the flattened corpses and swung it, completely disregarding the fact that the sword was covered with flesh and blood. Eight was overcome by a feeling of sickness. He retched twice but vomited nothing except bile.


[User] Please transcribe this text

[Assistant] Just then, outside the city wall, a battle cry resounded again.


[User] Please transcribe this text

[Assistant] The beast had just terrified the enemies, but now that they saw it had left, and they had no idea what had happened up there on top of the city wall, they began to stir again. By his highness, you mean the prince of border town? He wiped his mouth and gasped.


[User] Please transcribe this text

[Assistant] Is there another Roland besides him? The black-haired girl asked, as she started to dig another weapon from a corpse. Eight immediately turned his head away from her.


[User] Please transcribe this text

[Assistant] But it takes at least three days to get here from Border Town.


[User] Please transcribe this text

[Assistant] How did his highness know about the noble's rebellion so quickly?


[User] Please transcribe this text

[Assistant] Eight swallowed hard and continued, and the beast just now. It's not a beast, but a witch that came here to save you. Her voice turned cold. I don't have time to answer all your questions. Just keep quiet.


[User] Please transcribe this text

[Assistant] As there was no flintlock blocking them this time, the enemies easily came up on top of the city wall.


[User] Please transcribe this text

[Assistant] When torches lit the place up again, they were surprised to find only a girl standing in front of them. Malicious laughter soon broke the silence. Eight immediately knew what they meant by the laughter. Stay alert. Don't give them any chance to play any dirty tricks on us. Don't worry, my lord. We'll take care of it, but later. When I'm done, I'll leave her to you. Hey, fine with me. Quick, quickly come back here to me.


[User] Please transcribe this text

[Assistant] H struggled to stand up, but what happened next made him stare in amazement.


[User] Please transcribe this text

[Assistant] A silver light chopped down and the laughter died abruptly.


[User] Please transcribe this text

[Assistant] The leader of the enemies was chopped into two pieces by the long sword. His shield and armor could not stop it at all. In fact, he didn't even see the girl strike.


[User] Please transcribe this text

[Assistant] When the two pieces of his body fell down with blood spurting out from the wounds, everyone's smile froze on their face. However, this was just the beginning.


[User] Please transcribe this text

[Assistant] The black-haired girl took one step forward and swept her sword.


[User] Please transcribe this text

[Assistant] Before eight could clearly see your movements. Hear the sound of flesh scraping and bones breakings.


[User] Please transcribe this text

[Assistant] The three men had no time to react as they were ripped open in their stomachs.


[User] Please transcribe this text

[Assistant] Their intestines fell out mixed with blood and spilled all over the ground. You ate open his mouth a little but didn't know what to say.


[User] Please transcribe this text

[Assistant] The girl looked back at him and said, Go find your surviving battle companions and gather on top of the wall. Someone will come to fetch you later.


[User] Please transcribe this text

[Assistant] With these words, she jumped directly off the city wall. The city wall here was 30 feet tall.


[User] Please transcribe this text

[Assistant] He endured all the pains in his body, fumbled to climb close to the edge of the wall and looked down. He saw the crowd down there turning into a meat grinder. The girl killed wherever she went.


[User] Please transcribe this text

[Assistant] She freely dashed among the crowd, with her weapon moving around smoothly in her hand.


[User] Please transcribe this text

[Assistant] She easily cut down the entire enemy who dared to stand in front of her, like the reaping of wheat. In less than a quarter of an hour, the enemy collapsed.


[User] Please transcribe this text

[Assistant] They'd never seen such a fierce opponent, who was faster than a snow wolf and stronger than a grizzly bear.


[User] Please transcribe this text

[Assistant] Facing her sword, nobody could react, dodge, or fight.


[User] Please transcribe this text

[Assistant] The nobleman hurriedly retreated and the siege was extinguished, merely by one person.


[User] Please transcribe this text

[Assistant] She followed the escaping crowd all the way, leaving a trail of blood behind her.


[User] Please transcribe this text

[Assistant] The stronghold castle was tightly besieged by the four families armies.


[User] Please transcribe this text

[Assistant] A dozen bonfires around the castle brightly lit the place up.


[User] Please transcribe this text

[Assistant] After a day and night of fighting, the second floor of the castle had already been conquered.


[User] Please transcribe this text

[Assistant] The honeysuckles now shrunk back to the very top of it and must be suffering from both hunger and fear up there.


[User] Please transcribe this text

[Assistant] Shock Med looked at this towering lord's castle and began to feel excited.


[User] Please transcribe this text

[Assistant] After his father died, he traveled all the way back to the territory from the King's City just to inherit the Earl's title, but now he had an even better chance.


[User] Please transcribe this text

[Assistant] Timothy mentioned in the secret letter that if he could take hold of Longsong's stronghold for the King's city, the King's army would come here when the snow melted to conquer the rebel king, Roland Wimbledon.


[User] Please transcribe this text

[Assistant] Once the rebel king was eliminated, Timothy would probably let him govern the western region.


[User] Please transcribe this text

[Assistant] Together with the territory of Earl Honeysuckle, Jacques would have both the land and title of a duke. Duke Med. What a wonderful name. This castle would also become my residence.


[User] Please transcribe this text

[Assistant] My lord, the sixth platoon has come back down. A knight came and reported, and they said they heard a lot less flintlock sounds from the enemy. Is it time to send iron armor platoons up there? Shockmed nodded and said, go and arrange it.


[User] Please transcribe this text

[Assistant] Iron armor platoons were specially developed to fight against flintlocks.


[User] Please transcribe this text

[Assistant] During the fighting, three or four soldiers formed a team. Two of them held shields made of wood with several layers of iron coating, which could cover them all up, and had holes for aiming and firing.


[User] Please transcribe this text

[Assistant] In order to make more shields like this, he made the tough decision to tear down a dozen knight's armors.


[User] Please transcribe this text

[Assistant] Of course, those thick iron-coated shields had a weakness, too.


[User] Please transcribe this text

[Assistant] As it was hard to carry and the whole team had to move slowly they would easily become targets for their enemies. Luckily, the honeysuckles can't hold out any longer. Shock sneered silently.


[User] Please transcribe this text

[Assistant] They reacted unexpectedly quickly and withdrew part of their soldiers and knights back into the castle before the four families armies arrived. And they had just only nearly a hundred men. It's impossible for them to fight along. Hard battle.


[User] Please transcribe this text

[Assistant] It'll probably take several days before Prince Roland receives the news.


[User] Please transcribe this text

[Assistant] If I send Petrov's head as a gift to Roland Wimbledon, how will he look like?


[User] Please transcribe this text

[Assistant] My lord, the dock area in the west city seems strange. Reported a guard who was in charge of sealing off the street.


[User] Please transcribe this text

[Assistant] I've heard strange noises coming from there and sent two platoons to investigate but none of them came back. What? Shock Med said with a frown, you must have misheard it.


[User] Please transcribe this text

[Assistant] My lord, in this weather, anyone sleeping outside will be frozen to death. The guard insisted, and nobody can snore that loudly.


[User] Please transcribe this text

[Assistant] It must be a giant almost as tall as the city wall if they're really snoring sounds.


[User] Please transcribe this text

[Assistant] The Earl stared at him for a good long while and then turned away, shouting, Nightdowcan.


[User] Please transcribe this text

[Assistant] Summon your platoon and follow my guard to investigate the dock in the city.


[User] Please transcribe this text

[Assistant] Could you send someone else there? The young knight said, hesitating. My father urged me to enter the castle together with you.


[User] Please transcribe this text

[Assistant] It won't take you long to go to the dock, but I can assure you here it'll take a lot longer, the Earl said with a smile, and if the honeysuckle family surrenders, I'll wait for you here. Ah, fine.


[User] Please transcribe this text

[Assistant] After the night left with his guard, Jacques' facial expression turned stone cold.


[User] Please transcribe this text

[Assistant] You think you can receive the same credit by sending a son here to enter the castle with me? His majesty mentioned only me in the secret letter.


[User] Please transcribe this text

[Assistant] Before long, he heard the distant sound of gunfire coming from the west. Shock was instantly on the alert. What's that? The eldest son of the Maple family has no flintlock.


[User] Please transcribe this text

[Assistant] The guard said with his eyes widely opened, thousands of enemies are heading for the castle. You meant Roland Wimbledon's army? Shock raised his hand and slapped the guard on the face. Thousands of them?


[User] Please transcribe this text

[Assistant] If you dare to talk nonsense, I'll hang you up on the city gate.


[User] Please transcribe this text

[Assistant] My lord, they carry the flags of the kingdom of Greycastle. The guard did not dare to dodge.


[User] Please transcribe this text

[Assistant] He got down on one knee and continued, I wanted the squire of Knight Daukan to catch one or two enemies to get further information, but the moment they charged towards them, they were, were. What happened to them? The earl pressed. They were shot down by intense gunfire.


[User] Please transcribe this text

[Assistant] The Earl could tell from the guard's face that he seemed to have witnessed very terrifying scenes.


[User] Please transcribe this text

[Assistant] At that time, it looked like numerous fires popped out all of a sudden in the darkness and the cracking sounds never stopped.


[User] Please transcribe this text

[Assistant] 20 people only charged forward for less than 100 steps before they were all shot down and so did the horses.


[User] Please transcribe this text

[Assistant] The guard swallowed and went on, my lord, I've never seen such an intense gunfire.


[User] Please transcribe this text

[Assistant] If there weren't more than a thousand enemies, how could they eliminate the Knight's Platoon in just a blink? Where's the eldest son of the Maple family? He's run away.


[User] Please transcribe this text

[Assistant] The Earl sank, weak-kneed, into his chair and was completely confused. How is it possible?


[User] Please transcribe this text

[Assistant] The four families started to act from yesterday at noon and Prince Roland came to support Long Song Stronghold tonight.


[User] Please transcribe this text

[Assistant] If we count the messaging time, does this mean he needs only one day to come here from his town?


[User] Please transcribe this text

[Assistant] Even when sailing smoothly with the wind the entire time, they can't reach here that fast, let alone while carrying a thousand people.


[User] Please transcribe this text

[Assistant] To transport that many people, he needs at least a huge fleet. But according to the intelligence, Border Town has no ship in the winter. How could things turn out like this? No, I have to be calm in such an emergency.


[User] Please transcribe this text

[Assistant] Shock Med wiped the sweat from his forehead and thought that maybe the guard did talk nonsense.


[User] Please transcribe this text

[Assistant] Without torches, he could not see clearly how many enemies were there on the dock in the darkness when he was obviously shocked by their firing flintlocks. Flintlocks may seem powerful.


[User] Please transcribe this text

[Assistant] However, their shooting range was merely about 40 steps, and their reloading speed and hit rate were extremely low.


[User] Please transcribe this text

[Assistant] Assuming that every enemy is equipped with this kind of weapon, they won't be unbeatable for at most two dozen soldiers who can march abreast in the street leading to the castle.


[User] Please transcribe this text

[Assistant] If I inform the Wild Rose and the Wolf family now and gather all the knights, mercenary fighters, and guards to attack them after their first round of firing, maybe we could defeat them.


[User] Please transcribe this text

[Assistant] Unlike battles and fields, the poor efficiency in flintlock reloading is a fatal shortcoming in street fights. Dammit.


[User] Please transcribe this text

[Assistant] Go to invite Earl Wildrose and the Viscount of the Wolf family to come here and order all the men with flintlocks to block the entrance of the castle. Go!


[User] Please transcribe this text

[Assistant] Compared with crossbowmen, it's much faster to train flintlock soldiers. Let them block the enemies first. Their lives aren't worth much. If necessary, knights can tread on them.


[User] Please transcribe this text

[Assistant] To his great surprise, the head of his guards came back and reported they've already left the battalion with their men.


[User] Please transcribe this text

[Assistant] To besiege Petrov, the four families respectively held the four sides of the castle.


[User] Please transcribe this text

[Assistant] Shock did not expect that the other families received the news earlier than him. Now, the sounds of gunfire were nearing the castle.


[User] Please transcribe this text

[Assistant] As the guard had described, they were the deep rumble of dense drums, loud and clear across the growing mounds of snow. These bastards. Shock heart froze. He looked at the castle for the last time. He had no choice but to order a retreat. This time, only he and his guards had the chance to flee. His men in the castle were left behind. Outside the castle, the Earl was shocked by the scene.


[User] Please transcribe this text

[Assistant] The enemies were everywhere, and any knight who tried to break through the blockade would be shot down remorselessly. Their weapons were nothing like the flintlock sent by Timothy.


[User] Please transcribe this text

[Assistant] They could fire continuously and seemed to require no reloading. It was chaotic.


[User] Please transcribe this text

[Assistant] Members of the other three families were trapped, with the exception of Knight Dalcan, who had been the first to flee. My lord, what should we do? Send for the iron armor platoons. He cried out.


[User] Please transcribe this text

[Assistant] We'll forge behind them and we'll be covered by their iron shields.


[User] Please transcribe this text

[Assistant] After all the troubles, three iron armor platoons finally gathered and slowly moved forward holding their shields. The other knights found it and followed them, too. They knew that this might be their last chance to break through. Unfortunately, Jacques Med made a mistake.


[User] Please transcribe this text

[Assistant] When they were only 100 steps from their enemies, the latter's weapons exploded with dazzling bursts of fire.


[User] Please transcribe this text

[Assistant] The iron shields made to block lead balls were instantly riddled with bullets The men in the front holding the shields were killed


[User] Please transcribe this text

[Assistant] Splashing metal debris shot through bodies, stirring up a thick fog of blood behind the shields.


[User] Please transcribe this text

[Assistant] Before the Earl could call out the order of charge, he was cut down in a volley of fire.


[User] Please transcribe this text

[Assistant] That may be your impression. What do you mean? He ain't spending all his time at the tavern.


[User] Please transcribe this text

[Assistant] Well, choir, what'd you take for your rotten carcass? I'm an ocean of playing Yankee Doodle on your britches. Be off, you vermin, before I come at you like a steam engine. Chugga-chugga-chugga-chugga-chugga-chuck, and give you a


[User] Please transcribe this text

[Assistant] I'm very honored to kick off the, kick off the quarter. So what I'm going to be talking about today, or getting to eventually, is the idea, is that kind of a proposal for a system that would


[User] Please transcribe this text

[Assistant] People talk about and converse about and disclose, um, flaws in the published research record. Um, just to start off with some ground breaking news that will shock you. Um, millions of academic papers published, some of them are not correct.


[User] Please transcribe this text

[Assistant] Okay, so the question is how do we do things, sorry.


[User] Please transcribe this text

[Assistant] Our goal, sort of broadly speaking, right, is we want like more of these papers to be correct and fewer of them to be incorrect. Uhm, so we can actually learn things as scientists and as like consumers of research in the public, uhm, that are accurate.


[User] Please transcribe this text

[Assistant] So, uh, what I'm going to be talking about today, uhm, just is first to give some background about what's, what I really call like a crisis of credibility, uhm, in empirical social science right now. Uhm, some of you might have heard a lot about this, some of you might have heard almost nothing about it, so I'm going to kind of talk about this, tell like how it works, why is it that social science is in such bad shape right now.


[User] Please transcribe this text

[Assistant] Uhm, in my opinion anyway, uhm, I'm gonna talk about some existing efforts that are really great, that are trying to address, uhm, some of the problems I'm gonna talk about. Uhm, and then, uh, I'm gonna talk about kind of a reposed additional effort that I don't think will solve all problems, but I think might help solve some problems, uhm, that kind of plague the status quo, uhm, right now, and it can help accomplish that goal of more research is correct, uh, less research is incorrect.


[User] Please transcribe this text

[Assistant] I think the title of this really famous paper, this is from 2005, uhm, really kind of summarizes, uh, basically the state of the world we're in right now, uh, pretty accurately. Uhm, it's not just like some academic papers are false. Uhm, it is probably actually the case that a majority of academic papers are false. Uhm, this is bad.


[User] Please transcribe this text

[Assistant] Uhm, our job is not to be this way. We want to produce research that is correct. And there's, and I'm going to talk in a second about why this is, but this original paper in 2005 just did some basically careful math trying to understand, okay, based on the way that we know scientists do their work.


[User] Please transcribe this text

[Assistant] Uhm, and the way we know publishing works, which I'll get into in a second, uhm, what percent of papers are probably correct?


[User] Please transcribe this text

[Assistant] And the answer that arrived is, is most are not. Uhm, recently, uhm, this was just a few months ago, there was this really ambitious effort, uh, by, uhm, so folks at, uh, a non-profit called the Open Science Collaboration, based in the university, uh, or around the University of Virginia, and kind of science labs, psychology labs across the country, where they tried to kind of evaluate this claim empirically.


[User] Please transcribe this text

[Assistant] And they said, okay, let's go to some really, uh, prominent psychology journals. The journals that, like, important findings that have a lot of credibility associated with them. Let's go read a hundred psychology studies.


[User] Please transcribe this text

[Assistant] And then just replicate them exactly. So you read a study and it says, if you show people the color red, uhm, then they, you know, are more likely to use this word or something. Or if you, uhm, talk to people in a certain way about, let's say, another group, there'll be less prejudice towards that group.


[User] Please transcribe this text

[Assistant] So they said, let's just do exactly what these people did to see if the discovery they supposedly made or their theory is actually right. Just doing exactly what they did, replication.


[User] Please transcribe this text

[Assistant] Uhm, what they found when they just did the exact same studies that are in these prestigious journals is that most of the time, uhm, they could not actually replicate those studies. Uhm, so, consistent with this, uhm,


[User] Please transcribe this text

[Assistant] These theories from about 10 years ago, it actually does seem like


[User] Please transcribe this text

[Assistant] In social science journals right now, uhm, probably most of what we read is not true.


[User] Please transcribe this text

[Assistant] Uhm, so I would say this is about as worse, like we can't really be failing any more than this. And so like, why is this the case? Why are we failing? So there's this really great, I think it's like two or three page, uhm, little semi-satirical paper by


[User] Please transcribe this text

[Assistant] This, uhm, anonymous skeptic, uh, kind of critic of science, yeaah.


[User] Please transcribe this text

[Assistant] Uhm, so that I would definitely put under the heading of like, you know, in some sense uhm,


[User] Please transcribe this text

[Assistant] It's, uh, you know, it depends on the question you're trying to answer. Is it like, is the original paper wrong, or is it just like, not very useful in some other circumstances, we can't be sure.


[User] Please transcribe this text

[Assistant] Uhm, but that's kind of the evidence we have right now in any case. So uhm, some reasons why uhm, that might be the case, why it is that we might not be very good at producing accurate knowledge, uhm, are summarized in this very humorous pyramid, uhm, inspired by Dante's Inferno, uhm, by this anonymous critic of science, Neuroskeptic,


[User] Please transcribe this text

[Assistant] He calls the Nine Circles of Scientific Health. And this is kind of a chronicle of some of the practices that lead to this state of affairs. Uhm, and I'm going to talk about a couple of them in detail. Uhm, the first is something that has gotten a lot of, of uhm,


[User] Please transcribe this text

[Assistant] Uh, a lot of coverage recently, and this is the idea of what's called, uh, P-value phishing or, uh, P-hacking. Uhm, and here's how this works. Uh, when you do, for those of you who have not done a kind of empirical social science study, when you do a study, quite often what you do is you say, okay, I have some theory about the world.


[User] Please transcribe this text

[Assistant] Let's say this half of the room is randomly seated on this half. You guys get the blood pressure medicine. You guys get the sugar pill.


[User] Please transcribe this text

[Assistant] And now I'm going to measure your blood pressure and say, hey, it turns out the people that I gave the blood pressure medicine to, uhm, had slightly lower blood pressure. This is success. Okay. Well how do I know, you know, there's a 50-50 chance that one half of the room is going to have lower blood pressure than another. How do I know?


[User] Please transcribe this text

[Assistant] That really, this side of the room has such lower blood pressure, it's really unlikely that I just happened to put the low blood pressure people on this side. Well, we got this thing called a p-value, uh, which has rigorously stated the probability that if this blood pressure medicine didn't work,


[User] Please transcribe this text

[Assistant] Your blood pressure really would be as low as it is. So the idea is, I say, if you're a skeptic, you say, there's no way your pill works. I say, well, if you are right,


[User] Please transcribe this text

[Assistant] There's only a 1% chance that this side of the room, if we just randomly sorted people, we'd get people on this side of the room that have such lower blood pressure than this side. So that's a p-value. I'd say that's a p-value of .01.


[User] Please transcribe this text

[Assistant] And in the world of academia, uhm, generally speaking, the threshold we use for kind of over the bar of finding that is worthy of note, uhm, is a p-value of .05. Doesn't mean something is true if the p-value is below .05, it just means it's kind of clear to this arbitrary bar of this, this is the kind of evidence that a skeptic of your theory would not expect to see.


[User] Please transcribe this text

[Assistant] A paper, um, by, uh, some, uh, Simons and Nelson and Simmons in 2013. They had this idea where they say, well, you know, we, we see that what a lot of people are doing is they say, well, I know my colleagues, if I want to publish a paper, get a line on my CV, I know my colleagues want to see a p-value below .05. That's the bar.


[User] Please transcribe this text

[Assistant] And let's say I did this blood pressure study, I put you guys over here, you over here, and the p-value is like .06.


[User] Please transcribe this text

[Assistant] I say, man, that's not enough in the academic court of opinion to get my CV line, which I need.


[User] Please transcribe this text

[Assistant] So, what am I gonna do? Uhm, well, what I'm gonna do, uhm, and what they show a lot of people seem to be doing,


[User] Please transcribe this text

[Assistant] Is they say, well, you know, maybe this blood pressure medicine doesn't work for everybody, uh, but maybe it works for, let's say, people under 30. And so I look at just the people under 30, okay, no, like the p-value there's like 0.2, so okay, well,


[User] Please transcribe this text

[Assistant] Maybe it works, like, just for men. Okay, no, it doesn't work for men, for women, and you keep looking and you say, well, there is this one...


[User] Please transcribe this text

[Assistant] I thought that if we just look at men under 30, we would see this effect. And indeed, when we look at that subgroup, there's an effect. Then you have a paper, pill reduces blood pressure for men under 30.


[User] Please transcribe this text

[Assistant] Okay, so uhm, this is the kind of thing that people suspected was going on, uhm, that you can kind of keep looking in the data until you get that significant effect, uhm, which is troublesome, I should say, because even if your blood pressure medicine worked for nobody, if you just keep on cutting the data in a lot of ways, you're gonna eventually find something like that.


[User] Please transcribe this text

[Assistant] So what they said is, well, you know, it's hard to in any particular case know this is going on, but at the level of the entire community, you can actually get some sense of it. And they found some pretty disturbing evidence of exactly this.


[User] Please transcribe this text

[Assistant] So on this graph, what you're seeing on the red line is, imagine that if we have these p-values, if, if, imagine that experiment I just described, this blood pressure medicine that doesn't work, is done like 10,000 times in every university in the country. And then we just collect all the p-values from all those honest experiments.


[User] Please transcribe this text

[Assistant] Uhm, if that blood pressure medicine doesn't work, what we're gonna see is basically, because it's a probability, just uniform distribution where just as many p-values at 0, at .01, basically 5% of the p-values will be below .05, 4% will be below .04, etc. Just uniform distribution.


[User] Please transcribe this text

[Assistant] Not everyone's gonna find that magic below .05, but more people are gonna find lower p-values. So that's that line in green. So basically, if we have some chance, if the kind of effect is real, uh, and we have some chance of detecting it, you're gonna see more p-values kind of sort of skewing such that there's more closer to zero.


[User] Please transcribe this text

[Assistant] But instead, when you look at a lot of literatures in psychology and even in medicine, what you find is the opposite.


[User] Please transcribe this text

[Assistant] Which makes no sense statistically. Uhm, and one of the only ways you can get it is if researchers are doing what I just described. If they say, well,


[User] Please transcribe this text

[Assistant] I have this data and I'm just gonna kind of like keep on looking at it in different ways.


[User] Please transcribe this text

[Assistant] And as soon as I manage to find a comparison that just ekes over that threshold below .05, I'm gonna say, aha! That's my paper! And then write a paper around that.


[User] Please transcribe this text

[Assistant] And when you collect all the p-values, uhm, and this, I think, I forget which this was, they have it on their blog, uh, Data Calata, you see, uhm, you see, you see evidence like this, yeaah.


[User] Please transcribe this text

[Assistant] And so really what we should see, in the worst case, is that kind of flat. And as we start to study true things, we should start to see more lower p-values. But what we see is the opposite of what you see, which makes no sense unless researchers are doing what they call p-hacking. And what's in blue is actually like an empirical, observed kind of p-curve, they call it, in some academic literature.


[User] Please transcribe this text

[Assistant] So, so that's not good. Uhm, so another, this is a recent study by one of my colleagues in the political economy group. So he looked at this kind of, just where you can sort of see this even happening more directly. So what they did is they said, you know, there's this cool competitive grant competition in psychology.


[User] Please transcribe this text

[Assistant] There's one in political science too, where what you can do is say, I have this idea for these experiments that can be done in Mechanical Turk. You have people who read some text, say if it changes their attitudes, say,


[User] Please transcribe this text

[Assistant] And what people would do is say, well, here's my idea for an experiment. I'm gonna show people, let's say, one of five different stories and then ask them questions about that story. Did you find it persuasive? Did you like it? Whatever else. Let's say. I'm really boring to study like that.


[User] Please transcribe this text

[Assistant] Uhm, and then, uhm, then they say when they send the proposal, like, here's what it is we're going to do, here's what it is we want to test. And in order to like, get the money for the proposal, you have to like, actually say what it is you want to do with the data.


[User] Please transcribe this text

[Assistant] Uhm, and in this particular case, how it worked is that the people actually, when you kind of win the proposal,


[User] Please transcribe this text

[Assistant] This group actually does the research for you. So when you win, they say, we're gonna go run your questions on our pool of subjects, uhm, and run your experiment for you, give you back the data, and then you get to go publish it. Pretty good, right? What that also means is that


[User] Please transcribe this text

[Assistant] These authors might, working with the people who ran this repository, Neil and his co-authors could go back and say, well let's look at these original plans researchers had.


[User] Please transcribe this text

[Assistant] And how often do they actually follow what they said? Oh, I'm going to look at say,


[User] Please transcribe this text

[Assistant] People like story A better than story B, uhm, and that's, that's my hypothesis. And what they find is that when you look in the papers, uhm, it looks pretty different than what people said they originally intended. Kind of consistent with that story of after the fact looking for the, kind of, good looking results.


[User] Please transcribe this text

[Assistant] So you find that, uhm, when you look at all the tests that are not reported, that people planned, they said to do, uhm, those usually are not significant. They don't meet that magic .05 threshold. But there's other ways of looking at the data, like, well, if we look at this other question that I asked, and I look at this other group,


[User] Please transcribe this text

[Assistant] Then I do see something, and that's what makes it into the final paper. And so you can actually see individual studies doing this, uhm, systematically.


[User] Please transcribe this text

[Assistant] So this is kind of a problem. Uhm, if you're, if you're already depressed, it'll make you a little more depressed. So this sort of is sort of dishonest in some way at the level of individuals. So we say, oh, all along I thought this. Uhm, now imagine that everybody actually is honest. Uhm, we're just lazy. Uhm, that's what the next, uhm, the next issue's about. This is called partial publication.


[User] Please transcribe this text

[Assistant] So, the idea here is, uhm, imagine that again there's those 10,000 universities around the country, all doing, uhm, their experiment. They all split the room in half, give half the people blood pressure medicine, half don't. As I said, 5% are gonna come up with the magic answer that this in fact does reduce blood pressure, even if it doesn't.


[User] Please transcribe this text

[Assistant] Okay, so we're gonna have these hundreds of people across the country who find this result, even though the vast majority of people are gonna get the truth and say, no, this probably doesn't work. Well,


[User] Please transcribe this text

[Assistant] How academic publishing works is when you have a crazy idea and it's wrong, then, you know, that happens all the time and you sort of move on to the next project.


[User] Please transcribe this text

[Assistant] So all the people who, they had this blood pressure medicine, they gave it to this side of the room, didn't do any better. Oh well, lots of things don't reduce blood pressure, let's move on.


[User] Please transcribe this text

[Assistant] But for the 5% of people that erroneously found that it did work,


[User] Please transcribe this text

[Assistant] And so, as a reader, as someone looking at academic journals, the only things people bother to write up and send in and that get published are the things that report a crazy idea that's true.


[User] Please transcribe this text

[Assistant] And all, when those same crazy ideas, when they're tested by lots of other people, when they're false, we never hear about it. So you as a reader might say, wow, there's 500 studies showing that this blood pressure medicine worked. That's great evidence that I should take it.


[User] Please transcribe this text

[Assistant] Really, what you're missing is the population of when thousands and thousands of other people didn't find that it worked, they didn't feel like they were going to spend the time to tell you that.


[User] Please transcribe this text

[Assistant] And you see this in the literature where when you look right at .05, there's way more p-values across lots and lots of papers that are below this .05 threshold. And as soon as you get above that, people are much less likely to publish their results.


[User] Please transcribe this text

[Assistant] So as a whole, even if we're all individually honest, that kind of laziness or taste for kind of interesting results or crazy ideas that are proven right instead of wrong means that collectively we kind of render dishonest answers.


[User] Please transcribe this text

[Assistant] So, this is all kind of troubling. These are at least some of the reasons. We don't have a good sense of, uhm, how widespread all these different practices are, or to what extent each of them contributes to the kind of, uhm, lack of replicability of literature in general.


[User] Please transcribe this text

[Assistant] But uhm, all of them probably do to some extent. So there's a lot of efforts right now uhm, and over the last 10 years, especially a lot of people have been thinking about what are ways in which


[User] Please transcribe this text

[Assistant] We can try to kind of combat this. Uhm, and I'm gonna just briefly mention a few ideas out there. Uhm, so one, this is one of the older ideas, the idea of making replication data available. So if you think about those studies where I showed people five different stories, asked them if they liked it, they were persuaded, and I just reported, well, you compare this one story to this other story on this one, you know, that selective reporting. If you have to post all your data,


[User] Please transcribe this text

[Assistant] People could hypothetically go back to it and say, wait a minute, what about all these other conditions? Or alternatively, wait, you said the blood pressure medicine worked for men under 30, but when I look at your whole data file, I see it didn't work for anybody else.


[User] Please transcribe this text

[Assistant] So the things that you might not want to mention in your journal article, let's say, if you have to post your whole replication data file, now all of a sudden, kind of another person, you're opened up to that transparency.


[User] Please transcribe this text

[Assistant] Another thing, another idea that's out there, the federal government actually recently instituted this, this idea of a registry of all studies. So the idea is, before I even start my blood pressure study, one of my


[User] Please transcribe this text

[Assistant] Uhm, 10,000 blood pressure studies across the country that we imagined. I have to actually go to the government and say, uhm, hey, by the way, I'm starting to do this study of this blood pressure medicine.


[User] Please transcribe this text

[Assistant] And if I ever want to get it published, or if I'm a drug company, get my drug approved, I have to show that before I even started the study, that I did that.


[User] Please transcribe this text

[Assistant] And that's a great idea, because that means when the 500 people go to the government and say, hey, I have this study showing that blood pressure medicine works, the government says, okay, well, before I recommend that people take this pill, I want to go look in the registry and say, how many other people tested this hypothesis I never heard from? And there's a lot of them.


[User] Please transcribe this text

[Assistant] Let's go like call them up and say, well, what did you find? How come we never heard from you? And see what the truth is.


[User] Please transcribe this text

[Assistant] Another idea is, uhm, what's called a pre-analysis plan. So just like there was that example of the, uhm, proposal made to the granting agency, people, there's a new norm that people should, before they do an experiment, actually say, here's the group that I'm gonna compare, here's the hypothesis that I'm testing.


[User] Please transcribe this text

[Assistant] And that way kind of guard against the perception that, oh, after the fact, you just picked, uhm, you just picked on the basis of which results look best, you can say, no, in advance, this is what I meant. And that way, it's much less likely that whatever significant result you're showing me, uhm, is just the result of looking for lots and lots and lots of potential chance effects.


[User] Please transcribe this text

[Assistant] And what I want to kind of talk about today though is the idea that this transparency is really good, but it's only half of the puzzle.


[User] Please transcribe this text

[Assistant] It is, it is kind of a necessary condition that we are able to understand more about the process behind published research.


[User] Please transcribe this text

[Assistant] So, so that like kind of mistakes, be they innocent or not so innocent, can be detected. But if the replication data for a paper, let's say, is posted, that means the mistakes or fraud or whatever else could be detected, but just the fact that it's out there in public doesn't actually do the detecting. Somebody actually has to go do that.


[User] Please transcribe this text

[Assistant] And so, uhm, what I'm gonna talk about is kind of a personal story, uh, related to this. Uhm, it's made me, uh, come up with some ideas for some new ways in which we could not just basically take all the transparent material that's out there and make sure that people are actually using it, kind of auditing the public record and, uh, sorry, the research record and seeing, okay, the studies that are out there, uhm, can we actually believe them?


[User] Please transcribe this text

[Assistant] And to do that, I'm going to talk about a personal story running that I had, or I think the lawyers told me I need to say, in my professional opinion, that I ran into the last thing in this pyramid, which is falsification of data.


[User] Please transcribe this text

[Assistant] About two or three years ago, uhm, the, or actually let me start earlier, uhm,


[User] Please transcribe this text

[Assistant] Those of you who were here in California, uhm, November 2008, there was this Proposition 8, which was a proposition banning


[User] Please transcribe this text

[Assistant] Why did you vote for Prop 8? And going door to door, having conversations with people, and they started to develop an approach that they thought was effective at reducing people's prejudice towards gay people and making them more open towards gay marriage. With the idea that this might be on the ballot again, these, you know, people they were talking to might have gay relatives or gay friends, trying to reduce kind of the prejudice that they thought was animating this vote.


[User] Please transcribe this text

[Assistant] Uh, and they did something, not just that was really a unique thing in politics, they did something that's even more unique among people working in politics, which is they said, well we have what we think is a good idea about how to accomplish something.


[User] Please transcribe this text

[Assistant] But we wanna know, is it actually a good idea, uhm, or is this just a bad, crazy idea?


[User] Please transcribe this text

[Assistant] And so they contacted academics and said, we want to have this thing that we, this thing we've developed, this kind of conversation where ultimately what they got to is asking people about the marriages that they had or the love in their life and trying to get them to see how gay people felt the same way about the marriages they wanted to have.


[User] Please transcribe this text

[Assistant] They wanted to see is this the kind of thing uhm, that actually works. And so they worked with a researcher in their neighborhood.


[User] Please transcribe this text

[Assistant] Uhm, his name is Mike Lacour, he was a graduate student at UCLA, uhm, just down the road from, uhm, their headquarters, and he did an experiment where, just like with the blood pressure medicine, instead of giving you the blood pressure medicine and you the sugar pill, he sent canvassers, he had the canvassers for the Los Angeles LGBT Center go door to door,


[User] Please transcribe this text

[Assistant] And deliver the blood pressure medicine, so to speak, that is have the conversation.


[User] Please transcribe this text

[Assistant] These, these, uh, canvases are going door to door. And now we have a treatment group randomly assigned that


[User] Please transcribe this text

[Assistant] You're identical at the beginning, but now one group has this kind of treatment of the conversation, one group has the control group of...


[User] Please transcribe this text

[Assistant] We know they, the same kind of people, they can open, they'll open the door of their home on a Saturday morning, but we're not gonna kind of give them this treatment.


[User] Please transcribe this text

[Assistant] And then he went back, allegedly, uhm, and surveyed these people to say, okay, well, what do you think about gay marriage now? Uhm, and the results of this were reported in Science, and they were really interesting, uhm, and I'm not gonna go into them too much, but the kind of banner finding, uh, was that a conversation with a gay person.


[User] Please transcribe this text

[Assistant] Uhm, increase support for, uhm, gay marriage very, very dramatically.


[User] Please transcribe this text

[Assistant] Uhm, and moreover, uhm, that this effect lasted. So, social science is full of apparent effects on people's attitudes towards people and groups and issues, but almost always what we find is that if you say, read some persuasive text today and you say, oh I, that really convinces me, you totally forget about that by the next day. Uhm, effects on attitudes decay very rapidly.


[User] Please transcribe this text

[Assistant] What was unique about this is something about conversations with a gay canvasser specifically were memorable, it seemed like forever. So a year later, the people who got that conversation were more supportive of gay marriage than the people who had the recycling conversation.


[User] Please transcribe this text

[Assistant] So uhm, this is a really cool study, uhm, and it kind of overturned, uhm, a lot of, uh, kind of what I had always believed and what kind of we, uh, being graduate students in social science always learn, that you can't really change people's attitudes with kind of a 10 minute conversation about something like this.


[User] Please transcribe this text

[Assistant] And so I said, something really special is going on here, and I wanna really understand why.


[User] Please transcribe this text

[Assistant] And so I sort of started working on a follow-up study and the, uhm, because of some other work that I had done, this group got in contact with me, uhm, and they wanted to do a follow-up study on some of their work about views towards transgender people. Uh, with, uh, gay marriage legal, they said, well, now the next frontier is the T in LGBT.


[User] Please transcribe this text

[Assistant] And so we started planning a study. Uh, and in the course of this, um, I wanted to look back at the original study to say, well, I need to kind of plan some things about how the study is going to go. I need to know how much precision to expect, what the response rates will be.


[User] Please transcribe this text

[Assistant] And in the course of looking through this data, this replication data that is required to be posted, I found some things that were pretty weird. So here's one example of I think the first thing that I found. So there is this one measure in this study called a feeling thermometer. And what a feeling thermometer is, is I ask you,


[User] Please transcribe this text

[Assistant] Imagine there's this thing called the feeling thermometer, and it goes from 0, which means very cold, to 100, which means very warm, and 50, which means like lukewarm and, you know, in the middle.


[User] Please transcribe this text

[Assistant] And I want to ask you on that scale, how do you feel about gay people?


[User] Please transcribe this text

[Assistant] And this is a notoriously terrible survey question, because it doesn't mean anything. Uhm, and so what you find is that when you ask people these questions, if you ask the same person the same question on Monday and Tuesday and Wednesday and Thursday, they like give you a totally different answer every day.


[User] Please transcribe this text

[Assistant] Because they have no idea how to take this thing, which is this kind of vague feeling they have about gay people, and map it onto this 0 to 100 scale. People are bad with numbers, and they're even worse at translating their feelings into numbers.


[User] Please transcribe this text

[Assistant] And so, I knew going in, okay, this is the kind of item that's gonna be really bad, you're gonna see a lot of people, and you also see on surveys total jokers all the time. People say, a hundred, one day, zero, the next, uhm, and there's all these kind of ways in which survey data are kind of crap that I'm used to seeing.


[User] Please transcribe this text

[Assistant] Uh, never, but what you actually saw is when you look at the people in the study, uhm, when you look at what people said when they were interviewed one week and then when they were interviewed only a few days later, uh, people's responses were very, very stable.


[User] Please transcribe this text

[Assistant] Uhm, and across something like 10,000 people who are supposedly in this study.


[User] Please transcribe this text

[Assistant] Nobody did those joker things that I know people do on surveys, where they give a really different answer.


[User] Please transcribe this text

[Assistant] And so, at the time, I sort of thought, okay, well, uhm, this is pretty weird. And I sort of started thinking about this and was like, okay, I might have...


[User] Please transcribe this text

[Assistant] Stumble, this paper, almost no, nothing from political science ever gets published in this journal Science, which is like the most prestigious academic journal, which this paper was.


[User] Please transcribe this text

[Assistant] Uhm, not only was it really prestigious in my field and changed my thinking a lot, it was influencing what real world people do. There was a referendum coming up in Ireland where uhm, they were trying to use this study to try to convince people to vote for this referendum and make it such that there was uhm, full marriage equality in Ireland. Uhm, there were also lots of other groups that were trying to adapt this technique door to door. I'll give you an example.


[User] Please transcribe this text

[Assistant] Climate groups were thinking, okay, one of the findings of the study was it has to be a gay person to have this long lasting effect.


[User] Please transcribe this text

[Assistant] Lots and lots of different people thinking. Even this morning on the train I got an email from somebody that said an animal rights group that was thinking about sort of using this before what happened happened. And so I realized this is not just the kind of thing where intellectually it's important to get this right. There's like


[User] Please transcribe this text

[Assistant] Uhm, and I might know something like really relevant to the thing they're spending their time on.


[User] Please transcribe this text

[Assistant] What I did, uh, is talking to, uhm, say, okay, I'm a grad student, I have this, like, suspicion, I'm not really sure about it, you know, I think I know how survey status data is supposed to look. So I talked to a bunch of senior colleagues in my field. We had conferences, I would give talks to other universities, and I would say, hey,


[User] Please transcribe this text

[Assistant] I found these patterns like, what do you think that, what do you think that I should do?


[User] Please transcribe this text

[Assistant] And what I heard from basically everyone was very surprising to me. And what they told me is that I should do nothing. They said,


[User] Please transcribe this text

[Assistant] Just move past it and pretend that you never saw this. Because it's not a clear case. You sort of have these suspicions. You have better things to do.


[User] Please transcribe this text

[Assistant] And what they told me is, well look, whenever you criticize the published record in this way, you know, there's a lot of papers criticizing other papers, and if you have sort of a new theory or a new explanation, you can show, hey, look at how smart I am, I have this new theory for this paper out there. That's a really respected normal thing in kind of academia, we just adversarial interactions.


[User] Please transcribe this text

[Assistant] But it's kind of really different when what you're doing or a critique you have


[User] Please transcribe this text

[Assistant] Some people, this person was on the academic job market at the same time as me, we weren't looking at any of the same jobs, but,


[User] Please transcribe this text

[Assistant] Some people said, well, you have this person at the same level as you, and you kind of criticize them in this way, it's going to look like you're kind of throwing elbows. So, you don't want to have that perception about you, kind of stay away from it.


[User] Please transcribe this text

[Assistant] Other people said, you know, here's the successful person, sorry, here's this person who's a grad student, you know, nobody likes it when you pick on grad students, and so what a tenured faculty member might be told, for example, is, hey, lay off, he's a grad student, you know, don't, don't pick on this poor, defenseless person. On the other hand, people said to me, they'll also say about people that are more senior in academia, they say,


[User] Please transcribe this text

[Assistant] You know, if you do this, it's just gonna look like you want attention for yourself. Here's this big study in science and you come along wanting part of the action, don't you?


[User] Please transcribe this text

[Assistant] And so everyone said, you know, if you do this, uhm, this is gonna have big negative ramifications for you, no matter how you think about it.


[User] Please transcribe this text

[Assistant] And they also said, what if you're wrong? And this was really the thing that weighed on my mind even more. I had this evidence that I sort of was worried about, but I wasn't really sure. And the last thing you want to do is tarnish someone's reputation or ruin their career, uhm, when they're kind of a promising smart person who I knew personally, uhm, if you're wrong.


[User] Please transcribe this text

[Assistant] So uhm, I sort of wasn't sure what to do about this. This is the kind of thing I like talked about, but not that actively. A lot of other things to do was in the back of my mind. But eventually it came time for us to actually do the follow-up study. And when we were trying to do it, one thing that happened is we were having difficulty kind of following, sort of getting as many people to do the surveys as he did.


[User] Please transcribe this text

[Assistant] And the original study. So I said, well, he had this vendor, uhm, you're not supposed to be able to read the email, so don't worry about it. He, he had this vendor that he used to allegedly recruit people to these surveys. So why don't I go talk to this vendor and ask the vendor, hey, uhm, can you do what you did for me for him? And I had this email that was forwarded for me, uhm, from a few people where he said, hey, here's what the vendor is gonna do.


[User] Please transcribe this text

[Assistant] And so I called up this vendor, I say, hey, there's this person, Jason Peterson, your vice president of sampling and panel recruitment. Uhm, can I talk to this person because he did this big study for UCLA. I'd like to do it too.


[User] Please transcribe this text

[Assistant] You know, whatever it is UCLA bought, that's what I need. Uhm, and they all said, you know, we're a really small company and no person with that name has ever worked here. That person does not exist. Uhm, and then, and we definitely don't offer any of the services like the ones described in this email.


[User] Please transcribe this text

[Assistant] Run like two lines of code and basically reproduce exactly the data set that allegedly created this paper.


[User] Please transcribe this text

[Assistant] So we saw that basically we could simulate data that looked exactly like, uhm, the posted data, uhm, in the paper, uhm, a little too easily. Uhm, and eventually, and so we sort of said, okay, now, uhm, we're pretty convinced there's something here.


[User] Please transcribe this text

[Assistant] Uhm, then, and one of the reasons why I'm so excited to talk to you all today and hear your thoughts on this, uhm, is, uhm, we, uh, uh, I, I was sort of like, what do I do next? Uhm, and my roommate actually, uh, was the one who gave me the idea for how I should think about this.


[User] Please transcribe this text

[Assistant] He is the product of the, uhm, University of Washington Computer Science Department. Uhm, CTO of a company called HashiCorp.


[User] Please transcribe this text

[Assistant] And he said, you know, we have this idea in computer science called responsible disclosure. And the idea is that if you say find out a way to hack Google, you don't just post that publicly. You send a private email to Google and you say, hey,


[User] Please transcribe this text

[Assistant] Before I, like, say that I found this bug publicly, I want to give you some time to fix it. I want to give you advance notice. And that way, kind of, vulnerability can't be exploited, so to speak. And then, after the fact, you're kind of transparent, and you say, after the bug is fixed, Google says, hey, this person found the bug, and here's this email chain we had, here's how it all went down, just so everybody knows.


[User] Please transcribe this text

[Assistant] And so he said, and he gave me some ideas about how we could follow those same principles here. So what we did is we said, okay,


[User] Please transcribe this text

[Assistant] The vulnerability, our knowledge that we figured this out, we realized that we had asked a lot of these kinds of questions of Michael before. We realized, okay, now that we really, really know,


[User] Please transcribe this text

[Assistant] We can't just go to him and say, hey we know you made up the data. Because we knew that there were lots of things he could do and he was just going to keep making other things up. We knew he really had to figure out the truth and so we had his, uhm,


[User] Please transcribe this text

[Assistant] Graduate advisor basically sit him down on Monday morning, uhm, and say, okay, you did this big thing, you supposedly got a million dollars in grants, ten thousand dollars, you know, survey responsible, blah, blah. Do you have like one receipt or like piece of paper or piece of data or email or?


[User] Please transcribe this text

[Assistant] And sent in a retraction, we sort of came forward and say, hey, this all just happened and we want to be transparent about the fact that we had a role in it, we had suspicions for a while, and sort of here's what we found.


[User] Please transcribe this text

[Assistant] So uhm, there was this retraction letter sent in, uhm, by the co-author of the, uhm, of the paper, uhm, and then we posted this, uh, uh, this on my website. We called irregularities, uhm, in the, in the dataset we found.


[User] Please transcribe this text

[Assistant] It then blew up all over the media. It was kind of a crazy thing. Uhm, and uhm,


[User] Please transcribe this text

[Assistant] One of the most salient takeaways from all of this media insanity that happened, uhm, is, is that to this day, I have fellow researchers come up to me all the time or email me all the time and say,


[User] Please transcribe this text

[Assistant] Uhm, not, oh, it was so nice you did that, or you shouldn't have done that, or whatever. The most common thing I hear is, hey, me too.


[User] Please transcribe this text

[Assistant] I have this paper I've always thought there was this big issue with, but I haven't known what to do about it. Uhm, and this is what social scientists call pluralistic ignorance. When everyone hopes that other people do something and thinks it should be okay to do something, but expects that people wouldn't like it if they do it themselves. So everyone thinks,


[User] Please transcribe this text

[Assistant] It would be nice, I would like it if other people told me that mistakes they saw on the published record, but they expect that everyone else would judge them negatively for it. So people are wrong about what other people think.


[User] Please transcribe this text

[Assistant] And uhm, this is so pervasive that there's actually this website, psych file drawer, where people are supposed to post these replications, where if I replicate a paper and find it doesn't work, I'm supposed to say, hey I did this just so you guys know. They're having these issues where people don't even want to say,


[User] Please transcribe this text

[Assistant] I did the same study and I didn't find the effect. People are ca- are sort of don't want to admit to doing that. So there's this environment in academia right now where people don't want to come forward.


[User] Please transcribe this text

[Assistant] So I think this is bad, and it puts us in the kind of top, uhm, circle of these nine circles of hell, uhm, which the author calls a limbo. And this is those of us who, at any given time, are not committing, uhm, one of the, uhm, sins below, but are watching it all and not doing anything about it.


[User] Please transcribe this text

[Assistant] And so there's a lot of us there, and uhm, what I've learned is a lot of people are there and don't want to be there.


[User] Please transcribe this text

[Assistant] And, and that's because this status quo means that, you know, researchers make mistakes, innocent or not, and we kind of maybe talk about it at conferences and


[User] Please transcribe this text

[Assistant] There's kind of this folk wisdom about the papers that kind of everybody knows are wrong or that nobody knows are wrong if they should know are wrong. Authors can't respond to it. We can't really look into this and the public doesn't really know. So all the people that fund our research fund these buildings. They never actually know what we know about what research can be believed and not.


[User] Please transcribe this text

[Assistant] The idea here being some way of legitim, legitimizing a process. Uhm, I don't know if it would have worked in my particular case, but in many of the cases that I hear about, legitimize a process that would allow people to say, I have this concern about this published study. For example, I think that they didn't report everything.


[User] Please transcribe this text

[Assistant] And I want to look into that. And to do that, we can't just like, exhort people. There's one thing we know from social science and...


[User] Please transcribe this text

[Assistant] Probably convinced you we don't actually know all that much. One thing we do know is you can't just tell people, hey, you should behave this way, and then hope they do. That will not work.