賽後心得-「看見你的聲音--語音辨識後修正」

前言

決定命運的，不是偶然而是選擇。--電視劇 金裝律師 (Suits)

過往的失意，鬱結為一股不具名的悲傷，慢慢地，蛻變為養分。大學後我開始注重自己的選擇，追尋自己的意志，探索自己要的是什麼，結果是--迎來戲劇性的大學生涯。

雲霄飛車，一次次跌宕起伏，大學生涯亦然。突然發現自己喜愛資訊領域，進入後突然被抓去比賽，小有成就卻慘敗，進入一股低潮。莫名的，就站在舞台上，獲頒那夢寐以求的銅獎…替我曾經覺得遙不可及的奧匹資訊賽命題。

糊裡糊塗的倉促準備實習，進到我想進的公司 KKBOX。卻意外勾勒起落地又揚起的音符--「語音」與興致勃勃的我。於是隨性地訂下持續升學並研究語音辨識的目標。我想…很慶幸的，我並不是源自於對出社會的恐懼而升學，也並非背負家人的選擇而選擇，而是衷於我自己的想望，這是我最幸運的一件事。

去年十一月，推甄上台大資工所，並如願加入「多媒體資訊檢索實驗室」的語音組，此時的我像剛進入幼兒園，純真而好奇地探索每樣事物，懵懵懂懂地甚是純真，緩步地吸取新知。

我想，並沒有所謂的偶然，而是選擇決定了自己的命運吧！

緣起

實習、實驗室和課程，如此稀疏平常的生活一天天過去了。某天，兩位涉世未深的「碩 0 生」被指派參加玉山銀行舉辦的「看見你的聲音--語音辨識後修正」競賽，沒錯…其中一位就是我！

介紹

我們參加的是玉山銀行主辦的「看見你的聲音--語音辨識後修正」，看起來倒像綠豆糕，痾…不是！看起來很像「語音辨識」的主題，不過這塊領域長期是乏人問津，投入該領域的人並不多。事實上，他是「後修正」，意即做完語音辨識後，以「自然語言處理 (Natural Language Processing, NLP)」手法再進一步修正。

NLP 領域其實也並不多人，因此該比賽在 T-brain 平台上是最少人參與的，即便如此，仍有一百一十九隊參加！

銀行業近年來盛行導入智慧 AI 機器人，而該場比賽情境也十分雷同。玉山 APP 提供智慧化的語音服務，服務接受使用者的聲音後，經過 ASR (Automatic Speech Recognition) 模型辨識，產生出前十可能對應的語句--這些都是前半段「語音辨識」所屬的範疇。

賽事所做的就是後半段的「語音後修正」，依據語音辨識的結果，可利用不同候選句之間的關聯再次糾正，即為本次賽事的主要任務，不過…當然不僅如此而已，比賽較像是要求我們提供「正式對外服務」，我們需要架設 API Server 對外提供服務…！

每筆 Request 即為可能語句列表、時間和重試次數等等，當服務接受需求後，必須在一秒內回傳一句結果。然而…要命的來了，同時可能有十筆 Requests 席捲而來！更何況是在日益肥大的 NLP 領域達成這樣的豐功偉業，也因此一度「人性化」地在賽中放寬至兩秒，但仍舊是個很浮誇的需求。

小小不幸

何謂不幸？身為涉世未深的碩 0 學生，在賽事開始後一個月才參賽，以一個狀況外的鄉巴佬姿態誕生，如同新手逛街打怪，遇見地獄大魔王一般。我與隊友更是第一次參加 AI 相關賽事，憑藉隊友曾是 NLP 的過客，配合我在實習鍛鍊的後端能力，組成一支全面性俱佳的隊伍--CUDAOutOfMemory。

敝隊駕馭的電腦配備兩張 GeForce GTX 1080 Ti，而在我們初次與其相見歡的剎那，Out Of Memory 纏身--就那樣…戲謔性地成為了隊名XD

NLP Model

綜觀比賽本身，可以注意到不僅後端需求相當高，以語句本身的 baseline 亦是相當傑出，約略為 9.34 % CER (Character Error Rate)，這是經過玉山 ASR 辨識後的最佳結果。奠定在這基礎上，還想要更好的表現實屬不易，還得承受高強度的需求，只能說獲獎希望渺茫，藍瘦香菇啊……

我們選擇肥碩的 Transformers model，將「糾錯問題視為一種翻譯任務」展開一系列操作，添加額外文本 CLMAD、產生錯誤文本、困惑度 (Perplexity, PPL) 和 nlp-fluency。不幸地，在測試賽效果都十分有限。

零零總總的各方嘗試，於最終正式賽改進了 1 % 的 CER，成為唯一的 100% 使用 Supervised Learning 仍能獲獎的隊伍，真是萬幸啊XD

API Server

而我在這場比賽中，一肩扛起後端的重責。將這樣一個重大任務，交給一位初出茅廬，或許談不上是「後端工程師」的我來主掌，這隊也沒人才了…命在旦夕，得獎希望黯淡無光且渺茫啊…！

悵然若失、眼神空洞的….實驗室主機，一聲長嘆，然後一操再操。身為前六名中少數「超在意」後端的隊伍，也是少數從後端層面一次次加速，而達到賽事的要求，這也算是對實習一年來的成果驗收吧！

而接下來，就要面對一波波狀況，見招拆招了 :P

狀況一

想像一下，你在玩 Random Dice 這款骰子塔防遊戲。

身為左邊的玩家，你可愛的骰子們等級都太嫩，一隻敵人就應付不暇，更何況後面一條龍。是的，由於 NLP Model -- Transformers 的肥大，依據推論高達 2300 ms，遠超過主辦方要求，一個應付不暇，更何況一次來十個？

那麼，就得先針對問題本身思考，盡可能去蕪存菁。候選句全部都很重要嗎？這是第一個問題。人眼掃描後，可以發現越後面的語句越荒謬，在應付不暇的狀況下，其實第一句就已經堪用，保留了許多最重要的資訊了。

接下來，就得思考「模型」真有需要那麼「深度」嗎？由於訓練資料寡寡可數，大概六萬筆稱不上是大資料，Model level 太深，或許還有過度擬合 (Overfitting) 的風險，故可將層級壓低些。

如此一來，服務進展到單筆 700 ms 的狀態，可說是一大里程碑。但你可要知道「一個打十個」是多麽剽悍的事實。

狀況二

面對十個敵人，全數解決僅花一秒鐘。顯然地，隨著敵人增加，打倒全部所需的時間勢必也會線性成長，這對於服務本身也是相同的狀況。

因此，可從幾個層面著手：

• 排程--Queue vs. Concurrent
• 設備--CPU vs. GPU
• 拒絕滿分--「想守住越多，失去的越多」

拒絕滿分這項，其實離我們都很近。上了高中，若抱持著題題斃命的心態，數學就會使你一槍斃命。因此在以某種策略挑選題目一個個解出，才能達到最佳表現。而在本次賽事中，已知語句列表是按照排行榜排序的，因此若無暇處理它，那就回傳最佳的語句，就是一種損失最少的方式。

當前處理程序的方式屬於 佇列 (Queue) 排隊式的，程序其實和交通很相似，雪隧一但遭遇事故，整條國五就會呈現紫色地獄。如果有多條道路疏通，那就太好了！在處理要求 (Request) 就會面臨類似的窘境，但若以 Concurrent 的方式，則會讓整體服務速率都降低，若整體速率高於一秒，就更淒慘了…一筆也完成不了。

以 Concurrent 方式處理要求，我們實驗了 WSGI, Nginx 等正式部署工具，最終以 Flask 內建的 Thread 效能最佳，與其他兩項工具效能相差近三成。

Flask Thread 事實上是利用「快速切換」，在多筆 Request 間流轉、輪替，達成近似於「同時」處理的效果，而非真正意義上的「同時」，這是 Python 本身語言上的限制所致。總而言之，暫時以這樣的方式，配合 奴役 GPU 1080 和這三大策略，順利獲得以下的回應效能。

拒絕滿分這項，其實離我們都很近。上了高中，若抱持著題題斃命的心態，數學就會使你一槍斃命。因此在以某種策略挑選題目一個個解出，才能達到最佳表現。而在本次賽事中，已知語句列表是按照排行榜排序的，因此若無暇處理它，那就回傳最佳的語句，就是一種損失最少的方式。

Response Time Graph 會記載「何時」收到的 Request，每筆 Request 間隔多久才回傳結果，可以很直觀地看到整體服務的狀況是否在時間內。很顯然地在上圖中隱隱約約不妙、悲劇溢於言表，多筆 Request 回應時間高於兩秒，這代表整體效能還是不夠快。

狀況三--Batch

同時處理大量需求，始終會因為「高速」輪替而犧牲掉變換的時間 (context switch)，換個角度思考，是不是可以設定一個時間，在這段時間內盡可能搜集 request，時間到時一口氣同時處理，那麼配合 GPU 平行處理的威力，肯定能提高效能的吧！？

結果找到了這項工具 ShannonAI/service-streamer 完成了這項豐功偉業，Throughput 5.7 / sec 提升至 14.5 / sec，效率為原本的 254 % 之多。啊…斯巴拉西 (素晴らしい)！！

測試

速度到位了，就得進到測試，這是個博大精深的學問…

而我們用了三種方式一一去確認我們服務的運作、速率，以及在高壓狀態下是否能照常服務：

• postman 使我們能檢測 API 服務是否正常運作
• grequest 這項 Python 套件則讓人可對同時傳送 request 客製化
• JMeter 模擬十名使用者輪替傳送 request

一項產品上線後能否正常維運，壓力測試是很重要的環節，而你要盡可能的「高壓」去欺凌你自己的服務，把自己當成「壞壞使用者」去設計一些奧客才會做的事在測試裡頭，長句、錯誤句等等，若服務能夠安然無恙的照常運作與輸出，那服務「可能」就沒問題了。

上圖為 JMeter 的測試曲線圖，這套工具是提供測試的開源軟體，能夠模擬多名使用者運用服務的承載狀況。而從圖中能注意到紅、綠和藍色等等不同的曲線：

• 流通量 (Throughput)：每分鐘能處理的需求 (Request) 量。
• 平均時間 (Average)：每筆需求處理的平均時間 (毫秒)。
• 中位時間 (Median)：所有樣本 (sample) 按照處理時間排序後之中位數。

另外，可以設定「暖機」時間，在一個現實運作的服務中，流量也是逐步的上升與下降。我們可以仿照這樣的情境，設定幾秒後才達到最大用戶數，也因此綠色的流通量曲線並不是起初就維持高輸出，而是配合暖機時間逐步上升的。當然，JMeter 作為一個專業的測試工具，還有許多擬真情境的功能可以嘗試！這邊就不一一說明了。

總之！從這張圖中透露出服務能夠穩定的輸出某個定值，而單就當前的需求量而言，並不會影響到整體服務的效能，故能夠判定「可承受」比賽規格的服務需求。

架構圖

總之！從這張圖中透露出服務能夠穩定的輸出某個定值，而單就當前的需求量而言，並不會影響到整體服務的效能，故能夠判定「可承受」比賽規格的服務需求。

關於本隊的 API Server 是時候該做個總結了，以上是本隊的系統架構圖，流程大致如下：

1. 玉山發出 POST Request
2. Flask Server 為每個 Request 各創建一個 Thread 處理之
3. 蒐集 150 毫秒以內的 Request，最大容量為 64 筆
4. 每 150 毫秒 或超出容量上限，即 批次處理 (Batch)
5. 對於每筆 Request，檢查與判定，若符合規格就跑 Model，否則直接吐回 TOP1 的語句內容。這邊回傳給 Request 對應的 Thread
6. 回傳結果給玉山的伺服器

賽後檢討

流程大致上並無問題，也蠻順暢的。事後重新省視賽況和程式碼，認為以下幾點仍可改良：

• nvidia-docker： 運作時，有一段流程會不斷地查詢網路上的特定資源，以致於效率拖沓。因此，可以追蹤整體執行流程，將那些資源送進本機端維護並且定期更新，以本機端形式查閱，能提高約略三成的效能。
• multi-GPUs：由微軟公開的 ZeRO & DeepSpeed 加速技術，能夠將 NLP 模型龐大的參數攤分在若干個 GPU 上，能夠有效提升效能。
• Redis：是一種 Key-Value 的資料儲存形式，能夠高速查詢資料。因此我們能將超時的 Request 計算結果與唯一的任務代碼對應，那麼若再次重新發送就能夠接力計算，節省已逝去的時間。而本次使用的 service-streamer 工具中有提供 Redis 以 batch 方式加速的功能。
• Processes：常言「雞蛋不能放在同個籃子裡」，若以單個 Process 組織整套系統，當該筆 Process 出了狀況，將葬送整個服務。而以本次系統架構而論，即為單筆 Process 再分割出若干個 Thread 執行不同任務，因此日後得多在 multi-process 和 排程這塊琢磨，並需更加留意「例外」的處理形式。
• 程式碼維護：本次設定檔以及參數等工具不夠全面，只做了一半，如 Config 檔在 api 程式碼中就沒有使用，ArgumentParser 參數工具反倒是常更改的參數沒有寫進去。另一方面，由於程式碼撰寫頗為急促，故無暇顧及 "clean code"，這些都是未來可以改進的方向。

插曲——大災難

競賽有若干個階段，亦有兩梯次的測試賽以及正式賽。測試賽為期三天，在指定期間內每日晚間六點，準時轟炸你各位參賽者，一共四千題，完全比照正式賽之規格。而正式賽與測試賽唯一不同的在於連續日期從三天變為四天。

自詡為奴隸 (也許他並沒這樣說XD) 的實驗室電腦，很僥倖地在兩次測試賽長達六天內並未出任何狀況，盡心盡力地為我們付出，並穩定地繳交出兩萬四千題的答案。

然而同學間戲弄改編的名言：「關鍵時刻比衰小！」總是剎那間恍惚一瞬發生…正式賽第三天--我最忙碌的一天…發生了事故。當天早上，實驗室電腦們接一連二地發生狀況，或許是被攻擊，我們家的是自行關機，但這一重開機 CUDA 就出問題了，使得 GPU 1080 Ti 絲毫無用武之地…

左右觀察法--每個人都知曉的祕技，據聞是在考試時左看看、右看看就能獲取答案。而我使出了忍術「上下觀察法」，非常單純，如白雪般皎潔無瑕。僅僅是觀察位於自己前後兩支隊伍，前面 (第五名) 差距不小，而後面 (第七名) 也有著不容忽視的鴻溝，權衡之下，假設第三、第四天已 "baseline" 成績作結，綜合前兩天平均可以穩妥地取得第六名，那麼維持 "baseline" 採取最安全穩定的做法就萬事休矣了！

不久，迅速完成一個「無腦」服務，能夠在連線瞬間將 Request 內容列表第一句話吐回去，啊！真是完美！最終憑藉這樣穩定的服務度過比賽的下半餘生，並達成了領獎資格要求「低於 baseliine」之成績，以第六名作收，可喜可賀。

其實從這次事件中，可以學到兩件事：

一、不要太有自信

切忌，勿對服務的穩定性太有信心，必須戒慎恐懼好好關切他！本地端終究是有風險，除了遇到意外而終止程序外，當然還有「不可抗力」類型的機器罷工！上雲與地端都有各自的好處，但若能發展「監控」服務主動通知，並且有緊急應對的方式，才能應對服務的各種狀況。

二、「爛」方法其實很好

"Baseline" 什麼都不做--站在比賽以及服務的立場上，是相當切實的手法。比賽力求成效，而在已知能夠穩定獲取名次的狀況下，穩，再求精進才是上策，而勿追求卓越而功虧一簣，十分可惜。

服務依然以穩定為優先，而若全掛相當於 P0 事件，盡可能讓損失降低才是上策，因此若能夠緊急恢復出一個堪用版本，就幾乎只寫了一大半。能夠在緊急且資源有限的狀況下，盡可能地權衡局勢，致力於降低損失，會是提供服務時最重要的事情。

如何做得更好

這是我在賽後不斷思索的一件事，而從不同隊伍的做法中，領悟到幾件事情…

一、看清資料的本質

從官方給的六萬筆訓練資料中，若仔細觀察可以注意到有不少類似於新聞標題的語句：「行政院擬發放五倍券」，但也有很生活化的語句：「我想進行匯兌服務」。因此資料本身可以分成兩種取向：一般、新聞，並且很顯然的糾正這兩種不同類型的語句，應無法以相同的邏輯去糾正他。

除了語料源頭的分類以外，從測試資料本身亦可以做出不少變化。如同：「近年來股市迭宕不停」這樣的語句，從相似詞上看，「股市」或許容易被辨認為「不是」，每個語句會有多種相似詞彙，經過語音辨識後所辨識之語句，最有可能是「音對字不對」的狀況。

因此，我們可以建立「相似字詞典」，將單筆測資利用相似字詞典排列組合，塑造出更多測試資料。合理地推估六萬筆資料可擴增五倍、六倍以上，用以訓練模型的話，更可再進一步下降 1% 上下的 CER。

二、快，還要更快！

有很多加速技巧是沒碰過、聽過就不會知曉的，例如：微軟公開的 ONNX Runtime 是以 ONNX 格式儲存，能以其技術加速五倍以上的推理速度。亦或是攤分參數的技術，一樣是微軟開發的 ZeRO 以及 DeepSpeed，能將現有資源最大的活化。若能夠再配合本隊本次使用的 Batch 技術，那麼甚至可以一秒內同時應付二十筆 Requests 是很可能的。

結語

本次賽事我們兩位涉世未深的 碩 0 好夥伴費盡了心力，用了許多技術，難能可貴的能在第一次參與這樣的比賽，就搶到很前面的名次，獲得獎項，這也很超乎我們的預期。

不僅如此，更驗證了過去一年實習絕非虛度，技術上有深度進展，軟實力亦是。奠定在此基礎上，熟悉了整個系統維運與加速的程序，也學習到更多技巧可以運用，相信下次會更得心應手！

這次比較可惜的就是整體上過於倉促，沒有看透資料本身，能達到的成效就會有所囿限，加速上也有很多手法可以併用，希望下次能有類似的比賽能夠參與，也期許未來能提供更穩定且快速的系統，在四處都留下我們的蹤跡吧！

Preface

Our choices, rather than chance, determine our fate. — Suits (translated from the Chinese quotation)

Past disappointments once settled into a sadness I could not name. Little by little, that sadness became something I could grow from. After starting university, I began paying more attention to my own choices: following my interests and asking what I actually wanted. What followed was a rather eventful university life.

It felt like a roller coaster. I unexpectedly discovered how much I enjoyed computer science, entered the field, and was promptly swept into competitions. Some modest successes were followed by a crushing defeat and a difficult low. Then, somehow, I found myself onstage receiving the bronze award I had dreamed of. I even helped write problems for the informatics Olympiad, something that had once seemed far beyond my reach.

After a hurried, slightly bewildering internship search, I joined the company I had hoped to work for: KKBOX. There, speech technology caught my attention, with all its rising and falling notes. Almost on a whim, I decided to continue studying and research speech recognition. Looking back, I am grateful that I chose graduate school because it was what I wanted, rather than because I feared entering the workforce or was following my family's wishes. That freedom to choose has been one of my greatest pieces of good fortune.

Last November, I was admitted to NTU's graduate program in computer science and joined the speech group in the Multimedia Information Retrieval Laboratory, just as I had hoped. I felt like a child entering kindergarten: curious about everything, knowing very little, and slowly taking it all in.

Perhaps there is no such thing as coincidence after all. Perhaps our choices are what shape our lives.

How It Started

The days passed between my internship, the lab, and classes. Then two inexperienced students who had not yet officially started their master's degrees were assigned to enter E.SUN Bank's “看見你的聲音—語音辨識後修正” speech recognition post-correction competition. Yes, I was one of them!

The Competition

The competition was organized by E.SUN Bank. At first glance, its title seemed to suggest a speech recognition task. (My brain briefly went to mung-bean cakes instead. Never mind!) More precisely, though, it was about post-correction: using natural language processing (NLP) to correct text after speech recognition had already produced it. It was a relatively overlooked area, with few people working on it.

NLP was not exactly crowded either. This competition attracted the fewest participants on the T-brain platform, although that still meant 119 teams!

Banks had increasingly been introducing AI assistants, and the competition followed a similar scenario. E.SUN's app offered voice-based services. An automatic speech recognition (ASR) model took a user's speech and produced the ten most likely transcriptions. That was the first half of the process: speech recognition itself.

Our task was the second half. We could use relationships among those candidate transcriptions to produce a better final sentence. But the competition asked for more than an accurate model. We also had to run an API server that behaved like a service available to external users.

Each request contained the candidate sentences, timing information, a retry count, and other fields. The server had to return one sentence within one second. Here was the painful part: ten requests could arrive at once! Meeting that requirement with ever-growing NLP models was quite a challenge. The organizers eventually relaxed the limit to two seconds during the competition, which was a little more merciful, but still demanding.

An Unfortunate Start

What could go wrong? We were two students who had not even started our master's programs, joining a month after the competition began and barely knowing what was going on. It felt like a beginner setting out to fight a few monsters and immediately meeting the final boss. It was the first AI competition for both of us. My teammate had some prior experience with NLP, while I brought the backend skills I had picked up during my internship. Together, we formed a supposedly well-rounded team: CUDAOutOfMemory.

Our machine had two GeForce GTX 1080 Ti GPUs. The very first time we met it, we ran out of memory. And just like that, the error became our team name. XD

NLP Model

The backend requirements were demanding, and the recognition baseline was already strong: roughly 9.34% character error rate (CER), based on E.SUN's best ASR result. Improving on that while meeting the service constraints was no easy task. Our chances of winning did not look particularly bright.

We chose a hefty Transformer model and treated error correction as a translation task. We tried adding text from CLMAD, generating erroneous text, using perplexity (PPL), and applying nlp-fluency. Unfortunately, these approaches brought only limited improvements in the trial rounds.

After all those experiments, we improved CER by about 1% in the final competition. We also ended up as the only award-winning team to use supervised learning exclusively. We were very fortunate! XD

API Server

I took responsibility for the backend. Entrusting such a substantial task to someone as inexperienced as me, who might not even qualify as a backend engineer yet, probably says something about our staffing situation. Our prospects looked rather precarious!

I stared blankly at the lab machine, sighed, and put it through one experiment after another. Among the top six teams, we were one of the few who cared a great deal about the backend, repeatedly accelerating the service to meet the requirements. In a way, this was a practical test of what I had learned during a year of internship work.

Then came a series of problems to tackle, one at a time. :P

Challenge One: One Request Is Already Too Slow

Imagine playing the dice-based tower-defense game Random Dice.

You are the player on the left. Your little dice are still too weak to handle even one enemy, let alone the line behind it. That was our service: inference with the large Transformer model took about 2,300 ms, far beyond the competition's limit. If one request was already too much, what would happen when ten arrived together?

We first looked for work we could avoid. Did we really need every candidate sentence? A quick human inspection showed that the later candidates tended to become increasingly implausible. Under pressure, the first candidate was already a reasonable starting point and retained much of the most useful information.

Next, did the model really need to be so deep? We had only about 60,000 training examples, hardly a large dataset. A very deep model could also risk overfitting, so we reduced its depth.

These changes brought the processing time down to 700 ms per request, a significant milestone. Still, handling one request was very different from handling ten at once.

Challenge Two: Ten Requests at Once

Now imagine having to defeat ten enemies in a second. If each one takes time, the total naturally grows with the number of enemies. The same was true of our service.

We worked on three fronts:

• Scheduling: queuing versus concurrent request handling.
• Hardware: CPU versus GPU.
• Letting go of a perfect score: trying to save everything can mean losing more.

That last idea is familiar outside engineering too. On a high-school mathematics exam, insisting on solving every question can leave you with no time to finish anything. Choosing questions strategically is often a better way to maximize your score. Here, the candidate sentences were already ranked. If we could not process a request in time, returning its best candidate was a way to limit the damage.

Our service was using a queue, which reminded me of traffic. One accident in the Hsuehshan Tunnel can turn National Freeway No. 5 into a sea of congestion. More roads would help, but concurrent processing had its own cost: every request could become slower. If all of them exceeded the deadline, we might fail to return even one useful result.

We experimented with deployment setups involving WSGI and Nginx. In our tests, Flask's built-in threaded server performed best, with a difference of nearly 30% compared with the other two approaches.

My understanding at the time was that Flask's threads rapidly switched among requests, giving the effect of handling them together rather than executing all the Python work simultaneously. I attributed that limitation to Python's threading model. With that arrangement, our trusty 1080-class GPU, and the three strategies above, we obtained the following response-time results.

The response-time graph showed when each request arrived and how long it took to receive a result, making deadline failures easy to spot. Unfortunately, several requests still took more than two seconds. We were not fast enough yet.

Challenge Three: Batching

Rapidly switching among many requests also meant spending time on context switches. What if we instead collected requests over a short interval and processed them together when that interval ended? That would give the GPU's parallel processing capabilities more work to do at once.

We found ShannonAI/service-streamer, which let us do exactly that. Throughput rose from 5.7 to 14.5 requests per second, about 254% of the original throughput. Wonderful!

Testing

Once we had the speed, we needed to test the service. That turned out to be a subject all of its own.

We used three tools to check correctness, performance, and behavior under heavy load:

• Postman to check that the API worked as expected.
• grequests to customize concurrent requests from Python.
• JMeter to simulate ten users repeatedly sending requests.

Stress testing is an important part of finding out whether a service can keep running after launch. You have to be a deliberately troublesome user: send long sentences, malformed sentences, and the kinds of requests an especially demanding user might try. If the service continues to respond correctly, it might be ready.

The chart above came from JMeter, an open-source tool for simulating the load generated by multiple users. The red, green, blue, and other curves represented different measurements:

• Throughput: the number of requests processed per minute in this test.
• Average: the mean processing time per request, in milliseconds.
• Median: the middle processing time when the samples were sorted.

We could also configure a ramp-up period. Real services do not usually jump from zero traffic to maximum load immediately. By gradually increasing the number of users over several seconds, we could approximate that behavior. That is why the green throughput curve rose gradually rather than starting at its maximum. JMeter supports many other ways to simulate real conditions, but I will leave those for another time.

The important result was that the service sustained a stable output rate at the load we needed. That gave us evidence that it could handle the competition's request pattern.

System Architecture

Here is how our server worked:

1. E.SUN sent a POST request.
2. The Flask server created a thread to handle that request.
3. The batching layer collected requests for up to 150 ms, with a maximum batch size of 64.
4. When 150 ms elapsed or the batch reached its capacity, it was processed.
5. Each request was checked. Requests that met our processing criteria went through the model; the others received their top-ranked candidate directly. Each result was returned to the thread handling its request.
6. The server sent the result back to E.SUN.

Looking Back at the Backend

The overall flow worked reasonably smoothly. Reviewing the competition and code afterward, I saw several things we could improve:

• nvidia-docker: one part of the execution flow repeatedly queried a particular online resource, slowing things down. Tracing those queries and maintaining a periodically updated local copy could, by our estimate, improve performance by roughly 30%.
• Multiple GPUs: Microsoft's ZeRO and DeepSpeed suggested another direction: distributing a large NLP model's parameters across GPUs to make better use of the available hardware.
• Redis: a fast key-value store could associate a timed-out request's computed result with its unique task ID. A retry could then reuse the work instead of starting over. Service-streamer also offered Redis-based batching support.
• Processes: we should not put all our eggs in one basket. Our system used a single process with multiple threads, so a process failure could take down the entire service. I wanted to learn more about multiple processes and scheduling, and pay closer attention to exception handling.
• Code maintenance: configuration and argument handling were only half finished. The API code did not actually use the configuration file, while frequently changed parameters were missing from ArgumentParser. We had also written the code in a hurry, leaving little time to make it clean and maintainable. These were all areas to improve.

An Interlude: Disaster

The competition had several stages, including two trial rounds and the final round. Each trial lasted three days. At precisely 6 p.m. each day, the organizers bombarded our services with 4,000 requests under the same conditions as the final competition. The final round differed only in lasting four consecutive days instead of three.

Our hard-working lab machine survived all six days of the two trial rounds without a problem. It steadily returned answers to 24,000 requests.

Then the joking saying among my classmates, “At the crucial moment, it comes down to who has worse luck,” caught up with us. On the third day of the final round, my busiest day, something went wrong. Several lab machines had problems that morning; perhaps there had been an attack. Ours shut itself down. After rebooting, CUDA stopped working, leaving our 1080 Ti GPUs unable to help.

Everyone knows the dubious exam technique of looking left and right for answers. I instead used the perfectly innocent technique of looking up and down the leaderboard. The fifth-place team was comfortably ahead of us, and the seventh-place team was well behind. If we returned baseline results on days three and four, the average across all four days should still keep us in sixth place. The safest option was therefore to hold onto that baseline.

I quickly built a very simple service: as soon as a request arrived, it returned the first candidate sentence. Perfect! That stable fallback carried us through the rest of the competition. Our overall result still beat the baseline, meeting the award requirement, and we finished sixth.

I took away two lessons from that incident.

1. Do not be too confident.

A service's stability deserves constant attention. Local machines carry risks, from a stopped process to hardware failures beyond our control. Cloud and on-premises deployments each have advantages, but monitoring that alerts us to trouble, along with a way to respond quickly, is what helps us cope with incidents.

2. A “bad” method can be a good fallback.

Doing nothing beyond the baseline can be a very practical choice, both in a competition and in a live service. Once a reliable result is enough to secure a placing, keeping it stable before trying to improve it is sensible. Losing everything in pursuit of a better score would be a shame.

Stability matters for a service too. A complete outage is a P0 incident, so limiting the damage comes first. Restoring a usable version gets a substantial part of the recovery done. When time and resources are limited, assessing the situation and reducing losses are among the most important parts of operating a service.

How We Could Do Better

I kept thinking about this after the competition. Looking at other teams' approaches gave me a few ideas.

1. Understand the data itself.

Among the 60,000 training examples, some sentences looked like news headlines, such as “The Executive Yuan plans to issue quintuple stimulus vouchers.” Others were everyday requests, such as “I'd like to use the currency exchange service.” Those were two different kinds of language: news and ordinary conversation. It seemed unlikely that exactly the same correction strategy would suit both.

Beyond classifying the source text, we could also generate more variations. Take a sentence about the stock market fluctuating in recent years. The Mandarin word for “stock market,” 股市 (gǔshì), could be mistaken for 不是 (búshì, “is not”). Similar-sounding words create many possible errors, and an ASR output can easily get the sounds roughly right while choosing the wrong characters.

A dictionary of similar-sounding words could help us generate combinations from each example. I thought we might expand the 60,000 examples fivefold, sixfold, or more, then use that data to train the model. My estimate was that this might reduce CER by another 1% or so; we had not demonstrated that improvement in this competition.

2. Make it faster, then faster again!

There were many acceleration techniques I had not encountered yet. Microsoft's ONNX Runtime, for example, offered another route for running models in ONNX format; I was interested in the possibility of speeding inference up by more than five times. ZeRO and DeepSpeed were further options for making better use of multiple GPUs. Combined with our batching approach, I thought these techniques might let us handle twenty concurrent requests within a second. That was a direction to explore, rather than a result we had achieved.

Closing Thoughts

The two of us, still waiting to officially start our master's programs, put a great deal of effort into this competition and tried many techniques. Placing well enough to win an award in our first competition of this kind was far beyond what we had expected.

It also showed me that my year of internship work had not been wasted. I had grown in both technical and interpersonal skills. This experience helped me understand the process of running and accelerating a service, and introduced more techniques I could try. I expect to feel more prepared next time.

My main regret was how rushed everything had been. We had not examined the data closely enough, which limited what we could achieve. We also had more acceleration methods available than we managed to combine. I hope to enter a similar competition again, build faster and more reliable systems, and leave a few more traces of our work along the way!

• Repository: https://github.com/yuchen0515/2022-Competition-CUDAOutOfMemory
• YouTube: https://youtu.be/l-A8QkgyIxo
• Original article: https://medium.com/@mathlin/賽後心得-看見你的聲音-語音辨識後修正-f6d147ff9950