Chapter 2 Deepdive

2.1 Project: EMSX

Storm, Kafka, zookeeper

2.2 Project: Atlas

RabbitMQ, BAS

2.3 Project: PVM

Guido is pronounced like ‘widow’.

Garbage Collection/ Memory Pool/ Object Pool

2.4 Project: Google Ads Release Dashboard

2.5 Challenges

2.5.1 STL memory management

http://stackoverflow.com/questions/17871970/vectorstring-does-not-clear-memory-after-out-of-scope

Why does server memory footprint never reduce even if memory is released?

Most operating systems and libraries allocate memory in blocks/pages which the library code requests from the OS. Then, when the memory is freed it is usually returned to a free list for the process (for reuse) since this is more efficient and the task of returning the memory to the OS is relatively expensive, especially if it would be requested again later.

Sometimes the page will be returned if the allocation/deallocation is well behaved. However in general the heap for the process will become fragmented and since only complete pages at the end of the address space can be returned to the OS, the memory is often held by the process in its free list.

list/deque reduced memory but vector is memory hog!!! Strange!!!

[cling]$ #include <unistd.h>
[cling]$ getpagesize()
(int) 4096
[cling]$

2.5.2 Bond Floor Calculation

This is the present value of all the cash flows embedded in the convertible bond while neglecting any possible conversion. This is often called investment value. It excludes any income coming from the convertible’s equity option component. The bond floor is the value of the hybrid security when the call option is stripped out of the instrument set-up. To obtain this, the conversion ratio is put to zero in the calculations and the issuer call is ignored as well. The put is taken into account since it represents an investment value to the investor.

  • Situation

One day I got a ticket from one of our biggest customers(PIMCO), who decleared that our bond floor calculation is wrong in a very rare but possible situation(when the trade date is 3 b days before the maturity date). The results are inconsistent in some cases.

  • Target

I was required to fix the issue as soon as possible as they have business depending on this calculation.

  • Action

I was working on some other projects then, but I immeidately prioritized the tieket as the highest. I talked to my manager and surprisingly found it was not a new issue. It has been reported by another customer in the beginning of this year. But that customer didn’t require it to be fixed soon. And nobody in the team has resource to fix it. I tested our program and found there are actually two issues.

But how to fix the issue I am not 100% sure. The details are like this:

In real world, a bond cannot trade before (MTY - DaysToSettle). But when calculating bond floor, we manually set the first put date as MTY and remove call options, conversion options, dividend protection etc. In other words, we created a virtual bond, and this virtual bond have settlement day equal to maturity day.

We do the calculation by passing cash flow data to quant library. I did’t know how to pass the cash flow to quant library. The original code author has left company and there’s no document, and the variable names are very confusing and misleading sometimes. For example, loadDate, tradeDate, valuationDate, settlementDate have different meaning in different scenarios. We need to discount the cashflow twice. So on quantlib side, valuationDate is settlementDate; but from cvbapp side, valuationDate is the tradeDate. It is very difficult to understand the logic, especially there are bugs inside the code. I worked until the next day’s 1AM and finally understood the entire logic and also why the old code was wrong. Then the next day I made a conference call with Bin from Quants, Hitomi Customer representative, came up with 3 solutions:

1. when settlement day equal to MTY, pass 0 day and 0 cashflow
2. when settlement day equal to MTY, pass 180 day and half-year's coupon as cashflow(because it is when next put happens)
3. when settlement day equal to MTY, move the pseudo-MTY 1 day after the first put date and pass 1 day cash flow

After discussion, we decided to use the second option. I implemented and tested my solution immediately and had another meeting with my manager and teammates to explain my code logic and the old code logic. They were very satisfied with it and then approved I deployed to production. It took less than 24 hours.

  • Result

Two customers are very happy with my solution in so short time.

2.5.3 Deutsche Bank Escalation Case in Borland

2.5.4 postgresql idle session issue with SQLAlchemy + context manager + generator + return and yield

2.5.5 Conflict with Manager I

  • Situation

This January I was working on two projects atlashst and mtmfxsvc, for writing and reading Mark-to-Market price and FX rate respectively. One important feature is to support backfilling missing data. My manager also joined the meeting about how to design the system.

Priceget->mtmfxsvc (getmissingmtmVec)
priceget->atlashst (backfillMTM)
  • Target

As EOD team, I need to design a service to support backdate trade. More specifically, I need to provide a stable service to provice MTM price and FX rate for company’s clients. When there is data in Orcale, return to customers; if not, find it (in the current database, or ticker plant) and send back to customers.

  • Solution

We disagree on how to represent missing data(missing price and missing currencies, mtmcurrency and valuation currency).

Price is double, so I suggest NaN to represent the missing price. Because price could be 0 in some situation, for example, when we parties entered a swap, the price/value at initial time is 0. For currency, data type is int, so I suggest to use INT_MIN.

My manager wanted to use another bool column called isValid to represent if data is missing or not. I think this design is not simple enough. It need more bandwith because this is a client server application. To convince him, I send the following email to him:

Hi Colm,

There are some examples how to work with missing data in computer science by
Wikipedia, R, Python, and CS undergraduate text book. If there is some reason we
could not use it, Can you please give me a concrete example? Any DRQSs or source
code?
Wikipedia: "NaNs may be used to represent missing values in computations."
https://en.wikipedia.org/wiki/NaN
How missing data are represented in R? 
http://www.statmethods.net/input/missingdata.html
Text book Computer Systems: A Programmer's Perspective Section 2.4 Floating Point P105
How to work with missing data in Python Pandas?

Regards,
Fuheng
  • Result

Manager adopted my design. The service is running very effectively since then.

2.5.6 Conflict with Manager II - Automation of Relinking and Redeployment

S: Bloomberg use a static build system, so for each build, it will connect to all the dependent library statically. If there is dependency issue, you need to run another tools(llcalc twice) to solve it. The whole process is time consuming especially for big projects. Whenever there are code changes from project itself, or from dependent libraries, developers need to relink the code, which takes a lot of time. Our team have used the current build system for many years and they are very much used to it, and not realized it could be improved. When I come up with the idea of Jenkins, my manager said there is no resouce because we have many client tickets and projects to do, which have higher priority than the build tools. Because build tickets goes to developers/project owner/PWHO owner directly, so my manager did not realized how much time developers have spent on relinking/deploying old projects(some are twenty years old)

T: Reduce the build time and introduce CI into our development life cycle and at the same time maintain our current development and support speed

A: I collected how many internal tickets of rebuilds of our team 8. For each relinking and redeployment, it will take 5-6 hours, and especially the deployment, you have to do it gradually in several days.

2.5.7 Project Failed - Splunk Adoption

S: Our BAS services running with multithreads and it generates logs in a distributed way. It is hard to trace/debug.

T: Find a uniformed way to check log

A: Set up GUTs and Splunk

R: Due to the expensive price of Splunk, our company set a quota for each department and team. Due to the size of our log, we have reached our limit in two weeks. So we have to give up splunk, and find other ways.

2.5.8 Project Beyond Plan

  • PVM - exec removal

  • atlashst - multithread adaption, and distributed lock

pre optimization is the root of all evil

bottleneck is not here, is at priceget, so gived it up

One pnum one database

Database are managed by DBA team.

聊一聊分布式锁的设计
http://www.weizijun.cn/2016/03/17/%E8%81%8A%E4%B8%80%E8%81%8A%E5%88%86%E5%B8%83%E5%BC%8F%E9%94%81%E7%9A%84%E8%AE%BE%E8%AE%A1/

2.5.9 remove exec keyword from Python standard library

爬虫那些事儿– 简介
http://www.cnblogs.com/lovemdx/p/3189828.html

爬虫那些事儿–页面变化检测策略
http://www.cnblogs.com/lovemdx/p/3189984.html

ptmalloc,tcmalloc和jemalloc内存分配策略研究
https://www.owent.net/2013/07/ptmalloctcmalloc%E5%92%8Cjemalloc%E5%86%85%E5%AD%98%E5%88%86%E9%85%8D%E7%AD%96%E7%95%A5%E7%A0%94%E7%A9%B6.html

TCMalloc Allocator for STL http://stackoverflow.com/questions/5292994/tcmalloc-allocator-for-stl

TCMalloc Allocator for STL
http://quedig.com/questions/5292994/tcmalloc-allocator-for-stl/

STL有大量动态分配是真的,有可能影响性能也是真的,但通常都是有针对性解决方法的,真不必因噎废食.上了tcmalloc我也还是遇上过内存分配大幅影响性能的情况,通常就是一些能很容易预知vector大致长度的地方,reserve一把立竿见影,当然这个不仅是内存分配问题了.另,用deque代替vector通常是过度(且错误的)优化,也是碎片内存大户,vector resize代价真没这么大. again: 没跑过profile就去抓性能问题的都是扯.

https://www.zhihu.com/question/38225973

Redis采用不同内存分配器碎片率对比
http://blog.nosqlfan.com/html/3490.html

2.6 Amazon BQ

2.6.1 [Amazon 必考]Amazon Leadership Principles 亚马逊领导力准则

Leadership Principles,也就是领导力准则,不仅仅是几条用来鼓舞人心的口号,更是成就了Amazon特有公司文化的秘诀.不管是为新项目讨论创意、寻找解决客户问题的方案,还是面试求职者时,它都是Amazon员工的重要参考标准和行为准则.

    1. Customer Obsession 顾客至尚

Leaders start with the customer and work backwards. They work vigorously to earn and keep customer trust. Although leaders pay attention to competitors, they obsess over customers. 领导者从客户入手,再反向推动工作.他们努力工作,赢得并维系客户对他们的信任.虽然领导者会关注竞争对手,但是他们更关注客户 .

Deutsche Bank was one of the biggest customers of Borland when I was working there on 2009.(too old).
Bondfloor calculation

    1. Ownership 主人翁精神

Leaders are owners. They think long term and don’t sacrifice long-term value for short-term results. They act on behalf of the entire company, beyond just their own team. They never say “that’s not my job”. 领导者是主人翁.他们会从长远考虑,不会为了短期业绩而牺牲长期价值.他们不仅仅代表自己的团队,而且代表整个公司行事.他们绝不会说“那不是我的工作”.

    1. Invent and Simplify 创新简化

Leaders expect and require innovation and invention from their teams and always find ways to simplify. They are externally aware, look for new ideas from everywhere, and are not limited by “not invented here”. As we do new things, we accept that we may be misunderstood for long periods of time. 领导者期望并要求自己的团队进行创新和发明,并始终寻求使工作简化的方法.他们了解外界动态,四处寻找新的创意,并且不局限于“非我发明”的观念.当我们开展新事物时,我们要接受被长期误解的可能.

    1. Are Right, A Lot 决策正确

Leaders are right a lot. They have strong judgment and good instincts. They seek diverse perspectives and work to disconfirm their beliefs. 领导者在大多数情况下都能做出正确的决定.他们拥有卓越的业务判断能力和敏锐的直觉.他们寻求多样的视角,并挑战自己的观念.

Learn and Be Curious 好奇求知 Leaders are never done learning and always seek to improve themselves. They are curious about new possibilities and act to explore them. 领导者从不停止学习,并不断寻找机会以提升自己.领导者对各种可能性充满好奇并付于行动加以探索.

    1. Hire and Develop the Best 选贤育能

Leaders raise the performance bar with every hire and promotion. They recognize exceptional talent, and willingly move them throughout the organization. Leaders develop leaders and take seriously their role in coaching others. We work on behalf of our people to invent mechanisms for development like Career Choice. 领导者不断提升招聘和晋升员工的标准.他们表彰杰出的人才,并乐于在组织中通过轮岗磨砺他们.领导者培养领导人才,他们严肃地对待自己育才树人的职责.领导者从员工角度出发,创建职业发展机制.

    1. Insist on the Highest Standards 最高标准

Leaders have relentlessly high standards - many people may think these standards are unreasonably high. Leaders are continually raising the bar and driving their teams to deliver high quality products, services and processes. Leaders ensure that defects do not get sent down the line and that problems are fixed so they stay fixed. 领导者有着近乎严苛的高标准 — 这些标准在很多人看来可能高得不可理喻.领导者不断提高标准,激励自己的团队提供优质产品、服务和流程.领导者会确保任何问题不会蔓延,及时彻底解决问题并确保问题不再出现.

    1. Think Big 远见卓识

Thinking small is a self-fulfilling prophecy. Leaders create and communicate a bold direction that inspires results. They think differently and look around corners for ways to serve customers. 局限性思考只能带来局限性的结果.领导者大胆提出并阐明大局策略,由此激发良好的成果.他们从不同角度考虑问题,并广泛寻找服务客户的方式.

    1. Bias for Action 崇尚行动

Speed matters in business. Many decisions and actions are reversible and do not need extensive study. We value calculated risk taking. 速度对业务影响至关重要.很多决策和行动都可以改变,因此不需要进行过于广泛的推敲.我们提倡在深思熟虑前提下进行冒险.

    1. Frugality 勤俭节约

Accomplish more with less. Constraints breed resourcefulness, self-sufficiency and invention. There are no extra points for growing headcount, budget size or fixed expense. 力争以更少的投入实现更大的产出.勤俭节约可以让我们开动脑筋、自给自足并不断创新.增加人力、预算以及固定支出并不会为你赢得额外加分.

    1. Vocally self critical

Leaders do not believe their or their team’s body odor smells of perfume. Leaders come forward with problems or information, even when doing so is awkward or embarrassing. Leaders benchmark themselves and their teams against the best.

http://jesusgilhernandez.com/2014/11/25/amazon-10th-principle-vocally-self-critical/
Have you been tempted to cover up mistakes and bad news? It’s tempting to give the appearance that you or your team are progressing more quickly than you actually are, especially when reporting to upper management. It’s tempting to remain quiet while others are telling a rosy story or making promises that are nicer than reality.

As leaders, we should speak up and make sure the truth is heard even though we risk making our team (or ourselves) look bad. When you tell the truth despite its social repercussions, you encourage others to give you honest feedback and many times pre-empt the need for others to criticize you.

https://meritandgrace.com/succeeding-at-amazon-as-a-christian-part-5-of-6/

    1. Earn Trust of Others 赢得信任

Leaders listen attentively, speak candidly, and treat others respectfully. They are vocally self-critical, even when doing so is awkward or embarrassing. Leaders do not believe their or their team’s body odor smells of perfume. They benchmark themselves and their teams against the best. 领导者专注倾听,坦诚沟通,尊重他人.领导者敢于自我批评,即便这样做会令自己尴尬或难堪.他们并不认为自己或其团队总是对的.领导者会以最佳领导者和团队为标准来要求自己及其团队.

    1. Dive Deep 刨根问底

Leaders operate at all levels, stay connected to the details, audit frequently, and are skeptical when metrics and anecdote differ. No task is beneath them. 领导者深入各个环节,随时掌控细节,经常进行审核,当数据与传闻不一致时持有怀疑态度.领导者不会遗漏任何工作.

    1. Have Backbone; Disagree and Commit 敢于谏言,服从大局

Leaders are obligated to respectfully challenge decisions when they disagree, even when doing so is uncomfortable or exhausting. Leaders have conviction and are tenacious. They do not compromise for the sake of social cohesion. Once a decision is determined, they commit wholly. 领导者必须要能够不卑不亢地质疑他们无法苟同的决策,哪怕这样做让人心烦意乱,精疲力尽.领导者要信念坚定,矢志不移.他们不会为了保持一团和气而屈就妥协.一旦做出决定,他们就会全身心地致力于实现目标.

    1. Deliver Results 达成业绩

Leaders focus on the key inputs for their business and deliver them with the right quality and in a timely fashion. Despite setbacks, they rise to the occasion and never settle. 领导者会关注其业务的关键决定条件,确保工作质量并及时完成.尽管遭受挫折,领导者依然勇于面对挑战,从不气馁.

2.6.2 Sample Behavior Questions

  • Tell me about a time when you have been faced with a challenge where the best way forward or strategy to adopt was not “clear cut” (i.e. there were a number of possible solutions). How did you decide the best way forward?

    Settlement day and MTY day are equal, how to do the calculation?

    1. discuss with my own team
    2. discuss with quant team and cooperate/communicate with them face to face

    Bond floor calculation

    去年我在交易系统EOD组的时候,接到一个客户BOA的问题,我们的交易系统后端的report有问题,有些数字没有显示出来.BOA是我们最大最长期的客户之一,数据也是最多的几个客户之一.经过调查发现问题来自于risk部门,risk部门负责的服务risk attribution计算时间太长,需要10个小时以上的时间load我们的EOD ASCII data. 当时有几个解决方案: 提前risk calculation的时间,这是一个workaround

  • Give me an example of a time when something you tried to accomplish and failed.

    Splunk

  • Give me an example of when you showed initiative and took the lead.

    Jenkins CI, ADBP adaption
    Regression Test

  • Give me an example of a time when you motivated others.

I have an intern sitting near me called Abe Wheeler. I sometimes invite him a tea in the afteroon and talk

For a employee, I work very hard to motivate my teammates and even managers.

  • Tell me about a time when you delegated a project effectively.

    atlashist, mtmfx BAS, work with other 3 teams

  • Give me an example of a time when you used your fact-finding skills to solve a problem.

    STL memory issue
    Jenkins Integration

  • Describe a time when you significantly contributed to improving morale and productivity on your team. What were the underlying problems and their causes? How did you prevent them from negatively impacting the team in the future?

    Upgraded and streamlined the continuous integration in my team’s software development process.

  • What three things you are you working on to improve your overall effectiveness?

    More communication with manager, teammates and external stakeholders
    Business logic and legacy code, also keeping an eye on new tech and methodologies to improve at a minimum cost
    Taking notes every days, working diary every day
    Domain knowledge, existing code and logic

  • Give an example of a tough or critical piece of feedback you received. What was it and what did you do about it?

    After I changed ASCII from binary dump, risk team complain the new data requires more memeory. They got alarm for big customers like BOA. They are one of our external stakeholders. I just optimized it.

  • Describe a situation where you made an important business decision without consulting your manager. What was the situation and how did it turn out?

    Doesn’t apply to me.

  • Tell me about a time when you had to analyze facts quickly, define key issues, and respond immediately to a situation. What was the outcome?

    A big bank(BOA) - riskattribution memory issue
    ALL clients - cvbsinit works fine in beta but not in production IBM, get a core dump in allocate memory for a small string
    cvbreg priviledge issue All clients reported by a big investment management company(PIMCO)- bond floor calculation issue when settlement day equal to maturity date

  • Tell me about a time when you have worked against tight deadlines and didn’t have the time to consider all options before making a decision. How much time did you have? What approach did you take?

    Risk attribution memory issue: revert to old data, and at the same time find better solution

  • Give me an example of your most difficult customer interaction and how you worked through it. What was the outcome?

    Deutsche Bank and Borland

  • Tell me about a time a customer wanted one thing, but you felt they needed something else. How did you approach the situation, what were your actions and what was the end result?

  • Tell me about a time when you went above and beyond the call of duty for a customer. Why did you take the action you did? What was the outcome?

I recommend having in mind 2 good, concise examples of your work experience for each leadership principle below that our Software Development Engineers focus most on. Definitions for each principle is attached.

Ownership
Customer Obsession
Have Backbone; Disagree and Commit
Earn Trust
Deliver Results
Insist on the Highest Standards
Deal with Ambiguity
Bias for action
Invent and Simplify