My life as a Ph.D. student, S01E02: because nearning is the key!

(Wait, shouldn’t that word be “learning”?)

(And how many keys do we have in the lab?)

Ugh, I hate machine learning. Deep learning especially.

I met machine learning for the first time when I was a freshman. Andrew Ng (I don’t need to introduce this guy, I guess) came to Tsinghua and gave a lecture about his work. It was so fascinating.

But when I got to work with machine learning for the first time, it was not so pleasant. We had a machine learning course. The lecturer is a theorist, with high self-esteem. (He asked which book did we use for our theory course. When someone mentioned that classic book by Sipser, he said we should at least use Modern Approach!) The course was mostly about proving one bound after another, and now I can remember none of them! We did have fun looking at some machine learning problems from a nonlinear programming or even game theoretic point of view, though. The only algorithm mentioned in the course was (kernel-) SVM, and that was not without a ton of theoretic analysis.

And he hates deep learning. He mentioned it only once, in the concluding part of the course, only to downplay it with sarcasm. Are all machine learning theorists like that?

(BTW the TA seemed to be a super lazy otaku girl)

And then, after learning abstract things for the whole semester, the final course project was a concrete one – to solve one problem on Kaggle! I don’t even have a decent learning algorithm in my repository.

But, challenge accepted!

And the result – it did not give me frustration. It gave me cancer. My best score on the benchmark (which is a mediocre one) was achieved with bugged code. Only after the course project deadline was I able to get a better score with a correct program.

Those with the best results were doing intern at an alumni’s computer vision startup. Surely, you can’t hope to beat state-of-the-art deep neural networks implemented with a powerful framework deployed on a cluster with your crappy hand-written naive algorithm running on your laptop!

What’s after that? The next semester, when I found out that that computational biology course is all about machine learning, I quit immediately. (Seriously, discovering new interaction between proteins and existing drugs by doing text mining on existing literature sounds just ridiculous.)

Well, I don’t really hate machine learning itself. It really is powerful. What I don’t like is the way people work with it, and the hype.

I’m not saying that you have to give a theory to justify your method. Theory of neural networks is hard, and somehow they just work super good in reality, I understand that. I mean, just dumping everything into your network, tuning your network at random, and hoping that magic will happen, that definitely is not the correct attitude! But, my perception is that, this is how a lot of people are doing machine learning right now.

And people are well aware of it! In China we call that “炼丹(liàn dān)” (or TCM if you prefer that), and at last year’s 21ccc one speaker called that “alchemy” – see, we even have internationally accepted terminology for those sort of things!

And the hype – the number of people doing machine learning is too damn high! It is not rocket science, but neither is it for dumb people who know only how to liàn dān!

I’ve avoided coming in touch with machine learning thus far. But now, it is coming for me!

Because, nearning is the key, ahahahaha!

Computer graphics today is not like decades ago when people focused on rendering. Those problems are largely solved. It has become much broader. Citing words from a talk I attended at MSRA, today, computer graphics is about generating novel content from existing content, in the form of images, videos, 3D models, or even something non-visual like text or sound. It is about capturing human’s creativity. And how do you do that? Learning, of course!

So I started learning to do learning!

The past several days were spent getting the environment sorted – to get cuda working on my Arch Linux machine was a bit of a pain for a casual linux user – the proprietary NVIDIA driver keeps crashing so I had to use bumblebee for my intel/NVIDIA dual GPU laptop, and I had to workaround the incompatibilities between cuda 7.5 and gcc 6. But I guess you can hardly call that a trouble.

I’m using torch because – that’s what my cooperating labmate is using. The first thing to try? Guess it – it’s the MNIST dataset!

The whole procedure was like – put a convolution layer – then a pooling layer – then another pair of such layers – then 3 fully connected layers – then just dump in the data and sit there watching the error drop.

No! That does not feel good. Did I do anything? The days when I was hopelessly tweaking my bugged crappy hand-written naive learning code were like a joke.

But anyway, this still is a step forward.

If learning is the key, then what is the key to learning? There must be something deep (no, I don’t mean a deeper network) that distinguishes groundbreaking machine learning research from liàn dān, something that you have to use your brain to figure out. We shall see.

In remembrance of an old comrade | 记一位老战友

(A note on the title: I’m not sure whether this is a legitimate usage of the word “remembrance” but at least I know that “in memory of” is for the deceased.)

After all these years I finally heard from Tianyi Bai again, albeit in an unusual way. I noticed that a new follower on my twitter might personally know me, and that one turned out to be his girlfriend.

Are we friends? I doubt so. Despite having known each other from grade 6 and attending the same school for 6 years, we rarely meet and talk with each other. Even when we talked, it was often more academic than personal. And we have lost contact after our freshman year.

I barely have a friend, and he definitely has fewer than me! (He has more girlfriends, though.)

Maybe what really linked us it that we have always been in similar positions in our middle & high school years.

We were learning the pascal programming language when we first met.

learning_pascal

(Good old boys. Photo by Long Xue if I remembered correctly. BTW are you going to John Hopkins?)

And a year later, we were surprised to see each other at the opening ceremony of NEYC’s middle school division. We were both in the math specialty class (which to our sorrow has ceased to exist). There were two such classes and we were not classmates, though.

Our math teacher has never taught such a bunch of naughty math class boys (yes, we only had very few girls) before, and we gave him a lot of headache. We never followed the learning method he taught us, but by the end of grade 8 we emerged to be intellectually superior, constantly scoring top scores in math exams. And in a math class that pretty much means that you are at the top.

xlarge_jo70_420400029938118c

(A score report, from a period when two students, one from each math class, would be paired)

We were competing in maths and programming contests at that time. We both got first prize in provincial level junior division contests in informatics and mathematics by grade 7 and 8, respectively. Since then we’ve made frequent appearances on the front page of our school newspaper, and sometimes on billboards outside our school gate.

In our high school years we took somewhat different paths. I focused on informatics and he focused on mathematics. Finally he got a gold medal in CMO while my best results in NOI were two silver medals.

Actually Yao-class gave him an offer when he got that gold medal, before going to Yao-class became my goal. But he chose to study maths at Peking University.

We did have a chance to cooperate. He intended to compete for Shing-Tung Yau High School Mathematics Award (which has since been renamed) and needed an algorithm for a math problem.

201112301559429126

(Me and him in Sanya for the awarding ceremony)

We were not good at making noises. We were not who caught the most eyes. But we were the ones who beard the banner for NEYC class of 2012. If anyone of the students were going to do academics, that would be us.

He went to Peking University and I went to Thinghua. I heard that he ended up at ENS Paris, but otherwise we’ve lost contact after the first year.

Now I’ve got an e-mail from him.

I couldn’t say that I’m a friend of his, but I’m probably among the only ones who knows a thing or two about him personally. His family is poor. He lost his father at a young age and his mother does not even have a job. They relied on his uncle, I believe. Life is hard, and you can see that through his eyes.

He doesn’t talk much and doesn’t try to make friends. He does play Rubik’s cube (better than me) and I’ve seen him play StarCraft and Kiseki series, but those probably are all the things beyond his professional life. I think his poor life made him close-minded and feel inferior.

I might have always been whining that I did not enjoy my life, but for him, life is simply not enjoyable. Even when it comes to his (arguably) favorite thing, mathematics. He might have got more success and less frustration than me in high school science olympiads, but the road ahead only became more difficult.

He once complained to me about getting beaten constantly by his roommates who barely studied. As far as I know, he is not super clever. Perhaps no more than me. If he did achieve more, I’d say that is because he is more hardworking. But the problem with mathematics is that hard work does not always pay off. And what’s even worse for him, even if his work does pay off, he probably won’t make a lot of money.

Money really is a big thing, particularly if you don’t have it! When you don’t have it, it becomes the major consideration in your every decision. He went to ENS Paris presumably because he would be fully funded there. But his life only became more miserable – he did not know french! He doesn’t even make friends in China, how could he live there? (And here you have me who waste bulks of fund purchasing eroge! Definitely not well spent!)

Seriously, mathematics is only for those who are rich and can care less about his own life than the prosperity of human mind, or those who are just insanely clever. And you will also need a lot of determination when you are not achieving. As one who chose to do engineering, I always looked up to those who had the talent and dared to do theory.

He is talented but he is no Ramanujan, and he’s not met a Hardy. Perhaps mathematics is not the right thing for him to do. Does he still love mathematics after all these frustrations? Did he ever considered doing something else or was he even interested in anything else? I’m not sure.

He wrote in the e-mail that he does not want to work on fundamental theories anymore and intended to do statistics and something else more practically useful. His girlfriend told me that he just wanted to be teaching in a high school. What a waste. He deserved more.

Being one of the few people he can turn to, I fell obliged to give an answer. I don’t even know what to say at this point. Do I tell him to quit? I really hope that he can keep doing mathematics and give us some big results. I’ve kinda betrayed my childhood dream to be a scientist, I hope he won’t do the same. Do I tell him to continue? It may destroy him if he does not achieve. Do I tell him to be more open minded? I does not seem that he has much freedom to choose a life. Do I tell him to try to be positive and enjoy life? He does not have that luxury! Do I tell him to not worry about life and that I am willing to support him? How am I able support him in the first place?

Maybe life could have been better if we spent more time together. I could become more hardworking and motivated and he could become more open and have a better attitude by making a close friend.

I need to figure out the answer. Wish him the best.

My life as a Ph.D. student, S01E01

“… It’s also a fucked up city, but it’s awesome.”

That’s what my boss said about LA when were chatting, before I got the offer.

Honestly, I’ve never expected to see someone like Hao in the academia. At least not as a faculty. He is so unconventional. I still remembered when I saw someone sharing a link to his page on a Chinese SNS site two years ago. People were laughing with admiration.

At that time, I’ve just transferred to Yao-class and haven’t decided to do graphics and do a PhD yet. But now, two years later, I become his student. Life is strange.

For the first time, I’m in the United States.

LA definitely isn’t like anywhere I’ve been. With all those palm trees on the street, it’s like the tropics for me. Surprisingly the whether is not very hot. I’d say it is even cooler than my hometown Shenyang despite way down to the south in terms of latitude. The sun might be burning, but there is not much moisture in the air, so when in the shadows or when the wind is blowing, you don’t feel hot.

And, the sky is blue.

USC is to the south of the downtown. That piece of land is like the urban area of a Chinese county, just orders of magnitude larger, and the campus seems to be the only thing decent. It is about 1/3 the size of Tsinghua, with a lot of buildings packed tightly inside, yet it feels more spacious. The red bricks and white pillars reminds me of the american styled old campus of Tsinghua. Sadly we don’t have here a Jeffersonian Grand Auditorium with a big lawn.

I don’t miss home. I didn’t even have a jet lag and slept pretty well in the first night.

It seems that we have over 300 students from China coming to study CS in USC this fall, the vast majority of then being master’s students. For me, that figure is just overwhelming. Those master’s students are surely coming for the money.

With in years I’d be rolling in cash if I chose to do the same. I’m pretty sure I’m more qualified to earn that big sum of money than most of them. But I just don’t want to go to the IT industry. Or at least I want a more research oriented job in the industry. I’ve never self identified as a developer.

I don’t care about money. I don’t even care about making a super useful software and become popular. I shall only code for something fun.

My life as a Ph.D. student, S01E00

Here I’m finally starting to write about my life again. I can’t remember exactly when was the last time I did this.

Was it 8 years ago? Like pretty much everyone else, I was young and naive, had no accomplishments whatsoever, and had hopelessly unrealistic fantasies about future and about love. Even better, I was a problematic student who rarely did his homework.

Yet somehow that was still one of the best times in my life. I was full of passion and life was exiting. Over the years I won silver medal twice in National Olympiad in Informatics, got admitted into Tsinghua, and finally graduated from the prestigious Yao-class, but as I climbed all the way up the steep mountain to become one of what many perceive to be the best CS students in all of China, my life just kept getting worse.

You just can’t deny that the excellence of your peer gives you frustration, especially when you are near the top when the normal distribution of talents becomes like an exponential one. I felt overshadowed. I missed out narrowly on a gold medal in my final outing at NOI when I was 17, while some got gold medal at the age of 14. I barely got into school of software in Tsinghua which IMHO is an inferior institution, and failed the Yao-class entrance exam. It was a year later when I finally got into Yao-class by transfer. My classmates were publishing papers in top conferences, one of my roommate jumped 4 grades and started his Ph.D. program in MIT at the age of 18, while I haven’t achieved anything noteworthy in the academic world. And when you think that you are not good enough, enjoying life is the last thing you want to do.

I din’t enjoy delicious food. They just taste all the same. I din’t enjoy sleeping. I din’t dream and sleeping was boring. I didn’t enjoy music. Those popular songs were like nonsense. I haven’t been to a cinema for the past 7 years. I do watch some anime and play some games, but I was constantly thinking “Oh, this is so inappropriate”. I almost gave up photography as a hobby because I’ve ran out of inspirations.

I’ve never truly loved anything. I was not living my life, I was just keeping myself alive.

But this year, there has been something happening in my mind.

I always wanted to be a scientist, from the days when I was in kindergarten. As I grew up and got involved more in the academia, I realized the sad truth that although we can still somehow call ourselves “scientist”, most of us does not get a chance to discover something significant. I definitely won’t be the one to prove that P does not equal NP. Nor the contrary. Doing research is nothing prestigious, it is just another way to earn a living and probably not a decent one.

Despite all these, I still did not change my goal. I just want to find another approach. If you can’t do something significant and are ultimately gonna get buried in the huge archive of papers and get forgotten, why not try something cool? At the very least, not-so-useful but cool is better than not-so-useful and boring. And who said that cool things cannot be significant?

And by doing academics, you can actually see more opportunities outside academics than those who does not do academics.

After adopting this new way of looking at things, my life started to change.

Pure academic excellence weigh less and I’m not worried that much about not being good enough anymore. You might not be so good, but you can be different.

I started to live my live and looking for things that I truly loved. For the first time I’m thinking about doing something other than academics. I could even try to become an artist – not just a random one, but one who knows cutting edge technology and can create things other others cannot even think of.

And now I’m here in the University of Southern California, studying Computer Graphics.

This is a new life. Everything has changed. New places, new people, new things to do, and completely different way of life.

I still don’t have any girls though, but I don’t care.

Let’s see how the story is gonna unfold.