dvcs in big corporation

November, 14th 2011 DVCS in big Corporation

DVCS in big Corporation



Solutions● Centralization● Visualization

Challenges● Authentication● Authorization

About● Me● DVCS


Quick notes

http://www.slideshare.net/dchaffiol/dvcs-in-big-corporation


About : me


About : me on SO

A Lot Rep

Many times during the day

Every single day

ask@me

100K+


CVCSServer sideClient side


And then, a miracle:


DVCSServer sideClient side


Git on a client


Git on a client

eclipse


Reaction?

Not enthusiastic


Issues? Authentication.

Who is VonC?

LDAP

X41064


Issues? Communication


Issues? Publication


Centralization

Server


Centralization

itsvcprd git


Server


Server

MUTUALIZED


Server


Server: not root

Sudo apt-get install git


Server: not alone

Services are managed by root


Server: not in control

/usr/local content can change at any time


Help?

http://serverfault.com/questions/281810/how-to-install-packages-on-linux-or-solaris-on-non-default-paths


Recompile Everything


Recompile Everything: root


Recompile Everything: alone● Tailored services (ssh, ldap, https)


Recompile Everything: in control

Your own version of ~/usr/local


Manual recompilation?

Download sources



Configure./configure --prefix=${HULA}/@@NAMEVER@@ --with-lib=${HULL} --with-openssl --with-curl --with-expat --with-iconv=${HUL} --with-gitconfig=${HUL}/var/gitconfig --with-editor=vim --with-perl=${HULA}/perl/bin/perl --with-zlib=${HUL} --with-tcltk=no --with-python=${HULA}/python/bin/python

./configure --prefix=${HULA}/@@NAMEVER@@ --with-lib=${HULL} --with-openssl --with-curl --with-expat --with-iconv=${HUL} --with-gitconfig=${HUL}/var/gitconfig --with-editor=vim --with-perl=${HULA}/perl/bin/perl --with-zlib=${HUL} --with-tcltk=no --with-python=${HULA}/python/bin/python./configure --prefix=${HULA}/@@NAMEVER@@

--enable-ssl=shared --enable-ssl --with-ssl=${HUL}/ssl --enable-proxy --enable-proxy-connect --enable-proxy-ftp --enable-proxy-http --with-ldap --enable-ldap --enable-authnz-ldap --enable-authn-alias --with-apr=${HUL} --with-apr-util=${HUL} --enable-mods-shared=all --with-z=${HUL} @@WITHOUT_GNU_LD@@

./configure --prefix=${HULA}/@@NAMEVER@@ --enable-ssl=shared --enable-ssl --with-ssl=${HUL}/ssl --enable-proxy --enable-proxy-connect --enable-proxy-ftp --enable-proxy-http --with-ldap --enable-ldap --enable-authnz-ldap --enable-authn-alias --with-apr=${HUL} --with-apr-util=${HUL} --enable-mods-shared=all --with-z=${HUL} @@WITHOUT_GNU_LD@@

./configure --prefix=${HULS}/@@NAMEVER@@ --enable-shared --enable-static --with-zlib=${HUL} --with-ssl-engine=${HUL}/ssl --without-privsep-user --with-pid-dir=${HUL}/var/run --with-default-path=@@PATH@@ --with-privsep-path=${HUL}/var/empty @@WITHOUT_GNU_LD@@



Manual recompilation?● Make● Make install



Rinse and repeat

GitGit

Gcc 3.4.6openssl,libssh2,curl,libiconv,expat,libidn,zlibGcc 3.4.6openssl,libssh2,curl,libiconv,expat,libidn,zlib

opensshApache Http, lynxSubversion, Python, perl


=

+

32 libraries

14 applications

4 modules (Perl or ruby)


Manual Automated recompilation

https://github.com/VonC/compileEverything


You've got git.

Now What?


What is missing?Server sideClient side


Gitolite: authorization script

Repo1: user1, user2

Repo2: user2, user3

gl-auth-command

+=

Server side

Git command

Client side

Cmd output

https://github.com/sitaramc/gitolite


Gitolite: openssh

Repo1: user1, user2

Repo2: user2, user3

Server side

Git command

Client side

Cmd output

gl-auth-command

ssh


Gitolite: forced command

Command= "compileEverything/gitolite/bin/gl-auth-command bjensen",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsaAAAAB3NzaC1yc2EAAA...

~/.ssh/authorized_keys


Gitolite: not for users

Repo1: fisheye

Repo2: sonar

Server sideClient side

gl-auth-command

ssh

Repo1: user1

Repo2: user2


SSH is not enoughServer sideClient side

ssh gitolite


Git & “smart http”Server sideClient side

httpd

git-http-backend


Gitolite: httpd

gl-auth-command

Server side

Git command

Client side

Http answer

httpd

LDAP

+=

git-http-backend


Gitolite: LDAP alias

<AuthnProviderAlias ldap myldap> AuthLDAPBindDN cn=Manager,dc=example,dc=com AuthLDAPBindPassword secret AuthLDAPURL ldap://localhost:9011/dc=example,dc=com ?uid?sub?(objectClass=*)</AuthnProviderAlias>

Httpd.conf


Gitolite: REMOTE_USER

Httpd.conf

ScriptAlias /hgit/ compileEverything/gitolite/bin/gl-auth-command/ <Location /hgit> AuthName "LDAP authentication for ITSVC Smart HTTP Git repositories" AuthBasicProvider myldap Require valid-user AddHandler cgi-script cgi </Location>


Gitolite: https://itsvcprdgit:8453/hgit

# GitHttp on 8453<VirtualHost itsvcprdgit.world.company:8453> ServerName itsvcprdgit.world.company ServerAlias itsvcprdgit SetEnv GIT_PROJECT_ROOT /path/to/repositories SetEnv GIT_HTTP_EXPORT_ALL SetEnv GITOLITE_HTTP_HOME /home/auser/compileEverything

Httpd.conf


Httpd: multi-domain SSL certificateServer sideClient side

httpd

itsvcprdgit.world.company

itsvcprdgit

X509v3 extensions: X509v3 Subject Alternative Name: DNS:itsvcprdgit.world.company, DNS:itsvcprdgit


Are we there yet?Server sideClient side

ssh

httpd

gitolite


GitWeb


gitweb.cgi ?Server sideClient side

httpd

?

Gitweb.cgigl-auth-command


GitWeb: GL_USER

# finally the user name$ENV{GL_USER} = $cgi->remote_user || "gitweb";# now get gitolite stuff in...unshift @INC, $ENV{GL_BINDIR};require gitolite; gitolite -> import;

~/gitweb/gitweb.conf.pl


GitWeb: repo_rights()

$export_auth_hook = sub { my $repo = shift; return unless $repo =~ s/^\Q$projectroot\E\/?(.+)\.git$/$1/; # check for (at least) "R" permission my ($perm, $creator) = &repo_rights($repo); return ($perm =~ /R/);};



GitWeb: https://itsvcprdgit:8443/git

DocumentRoot compileEverything/gitweb Alias /git compileEverything/gitweb <Directory compileEverything/gitweb> AuthBasicProvider myldap AddHandler cgi-script cgi DirectoryIndex gitweb.cgi </Directory>

Httpd.conf


Are we there now?Server sideClient side

ssh

httpd

gitolite

gitweb


CGit


cgit.cgi ?Server sideClient side

httpd

cgit.cgigl-auth-command


CGit: repo_rights()

if ($request_uri ne "/cgit/" && $request_uri ne "/cgit/cgit.pl/") { (my $repo)=($path_info =~ /\/([^\/]+)/); my ($perm, $creator) = &repo_rights($repo); if ($perm =~ /R/) system("compileEverything/cgit/cgit.cgi"); else print " <h1>HTTP Status 403 - Access is denied</h1>\n"; }

~/cgit/cgit.pl


CGit: https://itsvcprdgit:8463/cgit

DocumentRoot compileEverything/cgit Alias /cgit compileEverything/cgit <Directory compileEverything/cgit> AuthBasicProvider myldap SetEnv GIT_PROJECT_ROOT=.../repositories AddHandler cgi-script .cgi .pl DirectoryIndex cgit.pl </Directory>

Httpd.conf


And now?Server sideClient side

ssh

httpd

gitweb

cgit

https://itsvcprdgit:8453/hgit

https://itsvcprdgit:8443/git

https://itsvcprdgit:8463/cgit


What do they want?Server sideClient side

ssh

httpd

gitweb

cgit

https://itsvc/hgit

https://itsvc/git

https://itsvc/cgit

NO PORT NUMBER

SHORT NAMES


Reverse ProxyServer sideClient side

ssh

httpd

gitweb

cgit

itsvc


NGinx: https://itsvc/xxx

location /hgit/ { proxy_pass https://itsvcprdgit.world.company:8453/hgit/;}location /git/ { proxy_pass https://itsvcprdgit.world.company:8443/git/;}location /cgit/ { proxy_pass https://itsvcprdgit.world.company:8463/cgit/;}

nginx.conf


There, there?Server sideClient side

ssh

httpd https://itsvc/hgit

https://itsvc/git

https://itsvc/cgit


What!?Server sideClient side


Issue1: authorname


Issue1: gitolite + hookServer sideClient side

gl-auth-commandPre-receive

hook


Issue1: pre-receive hookglog=`git log --format='%cn~%h~%s' $new --not --all`for cns in $glog ; do atLeastOneCommit=true echo branch $name: $cns cn=ècho $cns | cut -d~ -f1` hash=ècho $cns | cut -d~ -f2` subject=ècho $cns | cut -d~ -f3` if [ "$cn" = "$GL_USER" ]; then echo "one commit found with $GL_USER as committer name" exit 0 fidone


Issue1: pre-receive hook effect

remote: no commit with a committer name equals to 'bjensen', so this push is denied.

push


Issue2: Actual user on server

putty



Issue2: authorname on serverauser@vonc-VirtualBox:~/gitolite/demo$ ../../bin/git commit -m "default user on server"[master c694ed7] default user on server Committer: auser <auser@vonc-VirtualBox.(none)>Your name and email address were configuredautomatically based on your username and hostname. Please check that they are accurate. git config --global user.name "Your Name" git config --global user.email [email protected]


Issue2: putty+ git wrapper

Git wrapper


putty


alias agitBjensenItsvcprdgit='alias git="${H}/sbin/wgit u bjensen,[email protected],itsvcprdgit.world.company,bjensen"'

auser@vonc-VirtualBox:~$ git st[ bjensen,[email protected] for itsvcprdgit.world.company ]# On branch masternothing to commit (working directory clean)

Issue2: authorname on server

[ bjensen,[email protected] for itsvcprdgit.world.company ]


Finally, are we there?Server sideClient side

ssh

httpd

gitolite

gitweb cgit

Pre-receivehook

Gitwrapper


Conclusion: Server is hard


Conclusion: Application is hard


Conclusion: Big Corporation


Any questions?



If you need to introduce any tool in a big corporation, this presentation will help you be ware of the question you need to be prepare to answer.

This is a more Git-oriented presentation, but most of it equally applies to Mercurial.



Solutions● Centralization● Visualization

Challenges● Authentication● Authorization

About● Me● DVCS


Quick notes

http://www.slideshare.net/dchaffiol/dvcs-in-big-corporation


About : me

The opinions and elements in this presentations are mine and does not represent my current or former clients.


About : me on SO

A Lot Rep

Many times during the day

Every single day

ask@me

100K+


CVCSServer sideClient side


And then, a miracle:


DVCSServer sideClient side


Git on a client


Git on a client

eclipse


Reaction?

Not enthusiastic


Issues? Authentication.

Who is VonC?

LDAP

X41064


Issues? Communication


Issues? Publication


Centralization

Server


Centralization

itsvcprd git


Server


Server

MUTUALIZED


Server


Server: not root

Sudo apt-get install git


Server: not alone

Services are managed by root


Server: not in control

/usr/local content can change at any time


Help?

http://serverfault.com/questions/281810/how-to-install-packages-on-linux-or-solaris-on-non-default-paths


Recompile Everything


Recompile Everything: root


Recompile Everything: alone● Tailored services (ssh, ldap, https)


Recompile Everything: in control

Your own version of ~/usr/local



Download sources



Configure./configure --prefix=${HULA}/@@NAMEVER@@ --with-lib=${HULL} --with-openssl --with-curl --with-expat --with-iconv=${HUL} --with-gitconfig=${HUL}/var/gitconfig --with-editor=vim --with-perl=${HULA}/perl/bin/perl --with-zlib=${HUL} --with-tcltk=no --with-python=${HULA}/python/bin/python

./configure --prefix=${HULA}/@@NAMEVER@@ --with-lib=${HULL} --with-openssl --with-curl --with-expat --with-iconv=${HUL} --with-gitconfig=${HUL}/var/gitconfig --with-editor=vim --with-perl=${HULA}/perl/bin/perl --with-zlib=${HUL} --with-tcltk=no --with-python=${HULA}/python/bin/python./configure --prefix=${HULA}/@@NAMEVER@@

--enable-ssl=shared --enable-ssl --with-ssl=${HUL}/ssl --enable-proxy --enable-proxy-connect --enable-proxy-ftp --enable-proxy-http --with-ldap --enable-ldap --enable-authnz-ldap --enable-authn-alias --with-apr=${HUL} --with-apr-util=${HUL} --enable-mods-shared=all --with-z=${HUL} @@WITHOUT_GNU_LD@@

./configure --prefix=${HULA}/@@NAMEVER@@ --enable-ssl=shared --enable-ssl --with-ssl=${HUL}/ssl --enable-proxy --enable-proxy-connect --enable-proxy-ftp --enable-proxy-http --with-ldap --enable-ldap --enable-authnz-ldap --enable-authn-alias --with-apr=${HUL} --with-apr-util=${HUL} --enable-mods-shared=all --with-z=${HUL} @@WITHOUT_GNU_LD@@




Manual recompilation?● Make● Make install



Rinse and repeat

GitGit

Gcc 3.4.6openssl,libssh2,curl,libiconv,expat,libidn,zlibGcc 3.4.6openssl,libssh2,curl,libiconv,expat,libidn,zlib



=

+

32 libraries

14 applications

4 modules (Perl or ruby)


Manual Automated recompilation

https://github.com/VonC/compileEverything


You've got git.

Now What?


What is missing?Server sideClient side


Gitolite: authorization script

Repo1: user1, user2

Repo2: user2, user3

gl-auth-command

+=

Server side

Git command

Client side

Cmd output

https://github.com/sitaramc/gitolite


Gitolite: openssh

Repo1: user1, user2

Repo2: user2, user3

Server side

Git command

Client side

Cmd output

gl-auth-command

ssh


Gitolite: forced command

Command= "compileEverything/gitolite/bin/gl-auth-command bjensen",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsaAAAAB3NzaC1yc2EAAA...

~/.ssh/authorized_keys


Gitolite: not for users

Repo1: fisheye

Repo2: sonar


gl-auth-command

ssh

Repo1: user1

Repo2: user2


SSH is not enoughServer sideClient side

ssh gitolite


Git & “smart http”Server sideClient side

httpd

git-http-backend


Gitolite: httpd

gl-auth-command

Server side

Git command

Client side

Http answer

httpd

LDAP

+=

git-http-backend


Gitolite: LDAP alias

<AuthnProviderAlias ldap myldap> AuthLDAPBindDN cn=Manager,dc=example,dc=com AuthLDAPBindPassword secret AuthLDAPURL ldap://localhost:9011/dc=example,dc=com ?uid?sub?(objectClass=*)</AuthnProviderAlias>

Httpd.conf


Gitolite: REMOTE_USER

Httpd.conf

ScriptAlias /hgit/ compileEverything/gitolite/bin/gl-auth-command/ <Location /hgit> AuthName "LDAP authentication for ITSVC Smart HTTP Git repositories" AuthBasicProvider myldap Require valid-user AddHandler cgi-script cgi </Location>


Gitolite: https://itsvcprdgit:8453/hgit

# GitHttp on 8453<VirtualHost itsvcprdgit.world.company:8453> ServerName itsvcprdgit.world.company ServerAlias itsvcprdgit SetEnv GIT_PROJECT_ROOT /path/to/repositories SetEnv GIT_HTTP_EXPORT_ALL SetEnv GITOLITE_HTTP_HOME /home/auser/compileEverything

Httpd.conf


Httpd: multi-domain SSL certificateServer sideClient side

httpd

itsvcprdgit.world.company

itsvcprdgit

X509v3 extensions: X509v3 Subject Alternative Name: DNS:itsvcprdgit.world.company, DNS:itsvcprdgit


Are we there yet?Server sideClient side

ssh

httpd

gitolite


GitWeb


gitweb.cgi ?Server sideClient side

httpd

?

Gitweb.cgigl-auth-command


GitWeb: GL_USER

# finally the user name$ENV{GL_USER} = $cgi->remote_user || "gitweb";# now get gitolite stuff in...unshift @INC, $ENV{GL_BINDIR};require gitolite; gitolite -> import;
