UsingthePOSIXAPI

UsingthePOSIXAPI
Threads,realtimeandIPC

1
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

Thepthreadlibrary
InLinux,whenanewprocessiscreated,italreadycontainsa
thread,usedtoexecutethemain()function
Additionalthreadscanbecreatedusingthepthreadlibrary,which
ispartoftheClibrary
Ofcourseallthreadsinsideagivenprocesswillsharethesame
addressspace,thesamesetofopenfiles,etc.
Thepthreadlibraryalsoprovidethreadsynchronization
primitives:mutexesandconditions
Thispthreadlibraryhasitsownheader:pthread.h
Applicationsusingpthreadfunctioncallsshouldbeexplicitly
linkedwiththepthreadlibrary
gccoappapp.clpthread
2
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

Creatinganewthread
Thefunctiontocreateanewthreadispthread_create()
intpthread_create(pthread_t*thread,
pthread_attr_t*attr,
void*(*start_routine)(void*),
void*arg);

threadisapointertoapthread_tstructurethatwillbeinitialized
bythefunction.Later,thisstructurecanbeusedtoreferencethe
thread.
Attrisapointertoanoptionalstructurepthread_attr_t.This
structurecanbemanipulatedusingpthread_attr_*()
functions.Itcanbeusedtosetvariousattributesofthethreads
(detachpolicy,schedulingpolicy,etc.)
start_routineisthefunctionthatwillbeexecutedbythethread
argistheprivatedatapassedasargumenttothestart_routine
function
3
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

Creatinganewthread(2)

Address space

Address space

Thread

Thread

Thread

executing
main()

executing
main()

executing
func()

Process creation
using fork()

Thread creation using pthread_create()

4
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

Threadcreation,codesample
#include<pthread.h>
void*thread(void*data)
{
while(1){
printf(Helloworldfromthread);
}
}
intmain(void){
pthread_tth;
pthread_create(&th,NULL,thread,NULL);
return0;
}

5
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

Joinableanddetachedthreads
Whenthemain()functionexits,allthreadsoftheapplicationare
destroyed
Thepthread_join()functioncallcanbeusedtosuspendthe
executionofathreaduntilanotherthreadterminates.This
functionmustbecalledinordertoreleasetheressourcesused
bythethread,otherwiseitremainsaszombie.
Threadscanalsobedetached,inwhichcasetheybecome
independent.Thiscanbeachievedusing
Threadattributesatthreadcreation,using
pthread_attr_setdetachstate(&attr,
PTHREAD_CREATE_DETACHED);
pthread_detach(),passingthepthread_tstructureas
argument
6
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

Threadjoin,codesample
#include<pthread.h>
void*thread(void*data)
{
inti;
for(i=0;i<100;i++){
printf(Helloworldfromthread);
}
}
intmain(void){
pthread_tth;
pthread_create(&th,NULL,thread,NULL);
pthread_join(&th,NULL);
return0;
}
7
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

Threadcancelation
Itisalsopossibletocancelathreadfromanotherthreadusingthe
pthread_cancel()function,passingthepthread_tstructureof
thethreadtocancel.
#include<pthread.h>
void*thread(void*data)
{
while(1){
printf(Helloworldfromthread);
}
}
intmain(void){
pthread_tth;
pthread_create(&th,NULL,thread,NULL);
sleep(1);
pthread_cancel(&th);
pthread_join(&th,NULL);
return0;
}
8
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

pthreadmutexes(1)
Thepthreadlibraryprovidesamutualexclusionprimitive,the
pthread_mutex.
Declarationandinitializationofapthreadmutex
Solution1,atdefinitiontime
pthread_mutex_tlock=PTHREAD_MUTEX_INITIALIZER;
Solution2,atruntime
pthread_mutex_tlock;
...
pthread_mutex_init(&lock,NULL);
...
pthread_mutex_destroy(&lock);
Thesecondargumenttopthread_mutex_init()isasetof
mutexspecificattributes,intheformofapthread_mutexattr_t
structurethatcanbeinitializedandmanipulatedusing
pthread_mutexattr_*()functions.
9
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

pthreadmutexes(2)
Takethemutex
ret=pthread_mutex_lock(&lock);
Ifthemutexisalreadytakenbythecallingthreads,threepossible
behavioursdependingonthemutextype(definedatcreation
time)
Normal(fast)mutex:thefunctiondoesn'treturn,deadlock
Errorcheckingmutex:thefunctionreturnwiththeEDEADLK
error
Recursivemutex:thefunctionreturnswithsuccess

Releasethemutex
ret=pthread_mutex_unlock(&lock);
Trytotakethemutex
ret=pthread_mutex_trylock(&lock);
10
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

pthreadconditions
Conditionscanbeusedtosuspendathreaduntilacondition
becomestrue,assignaledbyanotherthread.
Initialization,staticordynamic
pthread_cond_tcond=PTHREAD_COND_INITIALIZER;
pthread_cond_tcond;
pthread_cond_init(&cond,NULL);

Waitforthecondition
pthread_cond_wait(&cond,&mutex)
Themutexwillbereleasedbeforewaitingandtakenagainafter
thewait
Signalingthecondition
Toonethreadwaiting,pthread_cond_signal(&cond);
Toallthreadswaiting,pthread_cond_broadcast(&cond);
11
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

pthreadconditionsexample
pthread_mutex_lock(&lock);

Receiver
side

while(is_queue_empty())
pthread_cond_wait(&cond,&lock);
/*Somethinginthequeue,
andwehavethemutex!*/
pthread_mutex_unlock(&lock);

Sender
side

pthread_mutex_lock(&lock);
/*Addsomethingtothequeue*/
pthread_mutex_unlock(&lock);
pthread_cond_signal(&cond);
12

FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

Managingrealtimepriorities
Seehttp://freeelectrons.com/docs/realtime/foranintroduction
CAPIAvailablethrough<sched.h>(seemansched.hfordetails)
sched_getscheduler,sched_setscheduler
Get/settheschedulingclassofaprocess
sched_getparam,sched_setparam
Get/setthepriorityofaprocess
sched_get_priority_max,sched_get_priority_min
Getthemaximum/minimumprioritiesallowedforascheduling
class.
sched_rr_get_interval
GetthecurrenttimesliceoftheSCHED_RRprocess
sched_yield
Yieldexecutiontoanotherprocess.
Canalsobemanipulatedfromscriptswiththechrtcommand.
13
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

POSIXsharedmemory(1)
Agreatwaytocommunicatebetweenprocesses
withoutgoingthroughexpensivesystemcalls.
Openasharedmemoryobject:
shm_fd=shm_open(acme,O_CREAT|O_RDWR,0666);
Azerosize/dev/shm/acmefileappears.
Setthesharedmemoryobjectsize
ftruncate(shm_fd,SHM_SIZE);
/dev/shm/acmeisnowlistedwiththespecifiedsize.
Iftheobjecthasalreadybeensizedbyanotherprocess,
youcangetitssizewiththefstatfunction.

14
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

POSIXsharedmemory(2)
Mapthesharedmemoryinprocessaddressspace:
addr=mmap(0,SHM_SIZE,PROT_WRITE,
MAP_SHARED,shm_fd,0);
Nowwehaveamemoryareawecanuse!
LockthesharedmemoryinRAM(bestforrealtimetasks):
mlock(addr,SHM_SIZE);
Usethesharedmemoryobject!
Otherprocessescanuseittoo.

15
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

POSIXsharedmemory(3)
Exiting
Unmapthesharedmemoryobject:
munmap(addr,SHM_SIZE);
Thisautomaticallyunlocksittoo.
Closeit:
close(shm_fd);
Removethesharedmemoryobject:
shm_unlink(acme);
Theobjectiseffectivelydeletedafterthelastcalltoshm_unlink.
Moredetailsinmanshm_open.

16
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

POSIXmessagequeues
DeterministicandefficientIPC.Seemanmqueue.h.
Advantagesforrealtimeapplications:
Preallocatedmessagebuffers
Messageswithpriority.
Amessagewithahigherpriorityisalwaysreceivedfirst.
Sendandreceivefunctionsaresynchronousbydefault.
Possibilitytosetawaittimeouttoavoidnondeterminism.
Supportasynchronousdeliverynotifications.

17
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

Creatingandopeningamessagequeue
Declarequeueattributes:
queue_attr.mq_maxmsg=16;
/*maxnumberofmessagesinqueue*/
queue_attr.mq_msgsize=128;
/*maxmessagesize*/
Openaqueue:
qd=mq_open(
/msg_queue,
OCREAT|O_RDWR,
0600,
&queue_attr);

/*queuename */
/*openingmode*/
/*permissions */

18
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

Postingamessage
Postingamessage:
#definePRIORITY3
charmsg[]=GoodbyeBill;
mqsend(qd,msg,strlen(msg),PRIORITY);
Closingthequeue:
mq_close(qd);
Caution:simplisticexamplecode.Shouldcheckreturnvalues.

19
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

Receivingamessage
Fromanotherapplication:
Openingthesharedmessagequeue:
qd=mq_open(/msg_queue,O_RDWR,
0600,NULL);
Waitingforamessage:
mq_receive(qd,text,buf,buf_size,&prio);
Closethequeue:
mq_close(qd);
Destroythequeue:
mq_unlink(/msg_queue);

20
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

POSIXsemaphores(1)
Resourcesforsharingresourcesbetweenthreadsor
processes.Seemansemaphore.h.
Namedsemaphores:
canbeusedbetweenunrelatedprocesses.
Unnamedsemaphores:canbeusedbetweenthreadsfrom
thesameprocess,orbyrelatedprocesses(parent/child).

21
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

POSIXsemaphores(2)
sem_open
Openand/orcreate
anamedsemaphore.
sem_close
Closeanamedsemaphore
sem_unlink
Destroyanamedsemaphore
sem_init
Initializeanunnamedsemaphore
sem_destroy
Destroyanunnamedsemaphore

sem_getvalue
Getcurrentsemaphorecount
sem_wait
Trytolockthesemaphore.
Waitotherwise.
sem_trywait
Justtriestolockthesemaphore,
butgivesupifthesemaphoreis
alreadylocked.
sem_post
Releasethesemaphore.

22
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

POSIXsignals
Signalsareamechanismtonotifyaprocessthatanevent
occured:expirationofatimer,completionofanasynchronous
I/Ooperation,oranykindofeventspecifictoyourapplication
Signalsarealsousedinternallybythesystemtotellaprocess
thatitmustbesuspended,restarted,stopped,thatishasdone
aninvalidmemoryreference,etc.
Eachsignalisidentifiedbyanumber:SIGSEGV,SIGKILL,
SIGUSR1,etc.
AnAPIisavailabletocatchsignals,waitforsignals,mask
signals,etc.
Seesignal(7)forageneraldescriptionofthesignal
mechanism
23
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

Registeringasignalhandler
Asignalhandlercanberegisteredusing
sighandler_tsignal(intsignum,sighandler_t
handler);
Thehandlerhasthefollowingprototype:voidhandler(int
signum)

intsigaction(intsignum,conststructsigaction
*act,structsigaction*oldact);
Thesigactionstructurecontainsthereferencetothehandler
Thehandlercanhavetwodifferentprototypes
voidhandler(intsignum)
voidhandler(intsignum,siginfo_t*info,void
*data)

Insidethehandlercode,onlysomefunctionscanbeused:only
theasyncsignalsafefunctions,asdocumentedbysignal(7).
24
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

Signalregistrationexample
#include<signal.h>
#include<assert.h>
#include<unistd.h>
#include<stdio.h>
voidmyhandler(intsignum)
{
printf("Signalcatched!\n");
}
intmain(void)
{
intret;
structsigactionaction={
.sa_handler=myhandler,
};

Fromthecommand
line,thesignalcanthen
besentusing
killUSR1PID

ret=sigaction(SIGUSR1,&action,NULL);
assert(ret==0);
while(1);
return0;
}

25

FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

Sendingasignal

Fromthecommandline,withthefamouskillcommand,
specifyingthePIDoftheprocesstowhichthesignalshouldbe
sent
Bydefault,killwillsendSIGTERM
AnothersignalcanbesentusingkillUSR1

POSIXprovidesafunctiontosendasignaltoaprocess
intkill(pid_tpid,intsig);
Inamultithreadprogram,thesignalwillbedeliveredtoanarbitrary
thread.Usetkill()tosendthesignaltoaspecificthread.

26
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

Signalsetsandtheirusage
Atypesigset_tisdefinedbyPOSIX,toholdasetofsignals
Thistypeismanipulatedthroughdifferentfunctions
sigemptyset()toemptythesetofsignals
sigaddset()toaddasignaltoaset
sigdelset()toremoveasignalfromaset
sigfillset()tofillthesetofsignalswithallsignals

Signalscanthenbeblockedorunblockedusing
sigprocmask(inthow,constsigset_t*set,sigset_t*oldset);

sigset_tarealsousedinmanyotherfunctions
sigaction()togivethelistofsignalsthatmustbeblockedduring
executionofthehandler
sigpending()togetthelistofpendingsignals
27
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

Waitingforsignals

2waysofwaitingforsignals:
sigwaitinfo()andsigtimedwait()towaitforblocked
signals(signalswhichremainpendinguntiltheyareprocessedby
athreadwaitingforthem.)
sigsuspend()toregisterasignalhandlerandsuspendthe
threaduntilthedeliveryofanunblockedsignal(whichare
deliveredwithoutwaitingforathreadtowaitforthem).

28
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

POSIXrealtimesignals
Regularsignals

POSIXsignals

Just2applicationsspecificsignals:
SIGUSR1andSIGUSR2

Wholerangeofapplicationspecific
signals:SIGRTMINtoSIGRTMAX

Nosignalpriorities

Prioritiesavailable.
Topprioritysignalsdeliveredfirst.

Signalscan'tcarryanyextra
information.
Signalscanbelost.Whenasignal
issentmultipletimes,thereceiver
willjustprocessoneinstance.

Possibletocarryextrainformation
inasignal.
Signalsarequeued.Allpending
signalsareprocessed:nosignalis
lost.

29
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

POSIXclocksandtimers
Comparedtostandard(BSD)timersinLinux
Possibilitytohavemorethan1timerperprocess.
Increasedprecision,uptonanosecondaccuracy
Timerexpirationcanbenotified
eitherwithasignalorwithathread.
Severalclocksavailable.

30
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

AvailablePOSIXclocks(1)
Definedin/usr/include/linux/time.h
CLOCK_REALTIME
Systemwideclockmeasuringthetimeinsecondsand
nanosecondssinceJan1,1970,00:00.Canbemodified.
Accuracy:1/HZ(1to10ms)
CLOCK_MONOTONIC
Systemwideclockmeasuringthetimeinsecondsand
nanosecondssincesystemboot.Cannotbemodified,
socanbeusedforaccuratetimemeasurement.
Accuracy:1/HZ

31
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

AvailablePOSIXclocks(2)
CLOCK_PROCESS_CPUTIME_ID
Measuresprocessuptime.1/HZaccuracy.Canbechanged.
CLOCK_THREAD_CPUTIME_ID
Same,butonlyforthecurrentthread.

32
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

Timemanagement
Functionsdefinedintime.h
clock_settime
Setthespecifiedclocktoavalue
clock_gettime
Readthevalueofagivenclock
clock_getres
Gettheresolutionofagivenclock.
Seemantime.handthemanualofeachofthesefunctions.

33
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

Usingtimers(1)
Functionsalsodefinedintime.h
clock_nanosleep
Suspendthecurrentthreadforthespecifiedtime,
usingaspecifiedclock.
nanosleep
Sameasclock_nanosleep,
usingtheCLOCK_REALTIMEclock.

34
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

Usingtimers(2)
timer_create
Createatimerbasedonagivenclock.
timer_delete
Deleteatimer
timer_settime
Armatimer.
timer_gettime
Accessthecurrentvalueofatimer.

35
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

Usinghighresolutiontimers
AvailableinLinuxsince2.6.21(onx86).
Nowavailableonmostsupportedplatforms.
Dependingonthehardwarecapabilities,
thisfeaturegivesmicrosecondornanosecondaccuracytothe
regularclocks(CLOCK_REALTIME,CLOCK_MONOTONIC).
Noneedtorecompileyourapplications!

36
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

AsynchronousI/O
HelpfultoimplementnonblockingI/O.
AllowstooverlapcomputetaskswithI/Oprocessing,
toincreasedeterminism.
Supportedfunctionality:
SendmultipleI/Orequestsatoncefromdifferentsources
CancelongoingI/Orequests
Waitforrequestcompletion
Inquirethestatusofarequest:completed,failed,orin
progress.
APIavailableinaio.h(manaio.hfordetails)

37
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

Compilinginstructions
Includes:nothingspecialtodo.
Availableinthestandardpath.
Libraries:linkwithlibrt
Example:
gcclrtorttestrttest.c

38
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

POSIXmanualpages
POSIXmanualpagesmaynotbeinstalledonyoursystem

OnDebianLinux,basedsystems,
tofindthenamesofthecorrespondingpackages:
aptcachesearchposix
Then,installthesepackagesasfollows:
aptgetinstallmanpagesposixmanpagesposixdev
Otherdistributionsshouldhavesimilarpackagenames.
Thesemanualpagesarealsoavailableonline:
http://www.opengroup.org/onlinepubs/009695399/idx/realtime.html
Youcanalmostconsiderthesemanualpagesasspecifications.
Thestandardcanalsobeaccessedon
http://www.unix.org/online.html(registrationrequired).
39
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

MoreinformationonthePOSIXinterface
ThePOSIXmanualpages
EmbeddedLinuxSystemDesignandDevelopment
P.Raghavan,A.Lad,S.Neelakandan,Auerbach,Dec.2005.
http://freeelectrons.com/redirect/elsddbook.html
Veryniceandclearcoverageonrealtimeprogramming
withthePOSIXinterface.Niceandusefulexamples.
Guidetorealtimeprogramming
http://www.phys.uu.nl/DU/unix/HTML/APS33DTE/TITLE.HTM
A11yearolddocument,withsomeDigitalUnixspecifics,
butstilluptodate(thankstostandards).

40
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

Relateddocuments

Allourtechnicalpresentations
onhttp://freeelectrons.com/docs
Linuxkernel
Devicedrivers
Architecturespecifics
EmbeddedLinuxsystemdevelopment
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

Howtohelp
Youcanhelpustoimproveandmaintainthisdocument...
Bysendingcorrections,suggestions,contributionsand
translations
Byaskingyourorganizationtoorderdevelopment,consulting
andtrainingservicesperformedbytheauthorsofthese
documents(seehttp://freeelectrons.com/).
Bysharingthisdocumentwithyourfriends,colleagues
andwiththelocalFreeSoftwarecommunity.
Byaddinglinksonyourwebsitetoouronlinematerials,
toincreasetheirvisibilityinsearchengineresults.

FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

Linuxkernel
Linuxdevicedrivers
Boardsupportcode
Mainstreamingkernelcode
Kerneldebugging
EmbeddedLinuxTraining
Allmaterialsreleasedwithafreelicense!
UnixandGNU/Linuxbasics
Linuxkernelanddriversdevelopment
RealtimeLinux,uClinux
Developmentandprofilingtools
Lightweighttoolsforembeddedsystems
Rootfilesystemcreation
Audioandmultimedia
Systemoptimization

FreeElectrons
Ourservices
CustomDevelopment
Systemintegration
EmbeddedLinuxdemosandprototypes
Systemoptimization
Applicationandinterfacedevelopment
Consultingandtechnicalsupport
Helpindecisionmaking
Systemarchitecture
Systemdesignandperformancereview
Developmenttoolandapplicationsupport
Investigatingissuesandfixingtoolbugs