tag:blogger.com,1999:blog-30532306167545164082024-03-19T00:30:42.823-04:00Joymon V/S CodeHere you can see software programming related articles,tips and sample code snippets.Joymonhttp://www.blogger.com/profile/12014569049635491702noreply@blogger.comBlogger635125tag:blogger.com,1999:blog-3053230616754516408.post-15593400827819971642024-03-12T21:30:00.011-04:002024-03-12T21:30:00.145-04:00Get Graph Id of folder or file in SharePoint Online UI without code<p>Often when we work with SharePoint Online via its APIs we will encouter situations to know the Graph id of documents or folders. Mainly when we want to debug our automation code or any code written aginst Graph API. The Graph Id is currently not displayed anywhere in the UI. This video shows how can we get the Graph Id of a document from UI or without writing code.</p><div class="separator" style="clear: both; text-align: center;"><iframe allowfullscreen="" class="BLOG_video_class" height="315" src="https://www.youtube.com/embed/3pjS7YTIcgQ" width="477" youtube-src-id="3pjS7YTIcgQ"></iframe></div><h2 style="text-align: left;">Urls used in the video</h2><p>https://vermarajneesh.sharepoint.com/sites/DataSite/_api/web/GetFolderByServerRelativeUrl('/sites/DataSite/Important%20Files/')</p><p>https://vermarajneesh.sharepoint.com/sites/DataSite/_api/web/GetFileByServerRelativeUrl('/sites/DataSite/Important%20Files/Project01/tracker.xlsx')/ListItemAllFields</p><p>https://vermarajneesh.sharepoint.com/sites/DataSite/_api/web/GetFileByServerRelativeUrl('/sites/DataSite/Important%20Files/Project01/tracker.xlsx')/ListItemAllFields/File?$select=VroomItemId</p><h2 style="text-align: left;">References</h2><p>VroomId added - <a href="https://devblogs.microsoft.com/microsoft365dev/new-sharepoint-csom-version-released-for-sharepoint-online-august-2020/">https://devblogs.microsoft.com/microsoft365dev/new-sharepoint-csom-version-released-for-sharepoint-online-august-2020/</a></p>Joymonhttp://www.blogger.com/profile/12014569049635491702noreply@blogger.com0tag:blogger.com,1999:blog-3053230616754516408.post-60750144852411410742024-02-12T21:00:00.001-05:002024-02-12T21:00:00.252-05:00[Video] Azure @ Enterprise - Adding reference line to App Insights charts<p>We may present KQL-generated charts to managers to show that the production is working fine, especially to show the system responds to web requests within time. If the managers are still hands-on they easily understand whether this report looks good. Sometimes particular projects may have higher performance standards than the industry standards. Also, there may be more complex metrics than just the request response time. In all these situations it is better to add a reference line to show when it is considered a problem.</p><p>Here is the way to add such a line in a short video</p><p><a href="https://youtube.com/shorts/Vpk7L6KY2Eo?si=z7Tkts_dsek4kPam">https://youtube.com/shorts/Vpk7L6KY2Eo?si=z7Tkts_dsek4kPam</a></p>Joymonhttp://www.blogger.com/profile/12014569049635491702noreply@blogger.com0tag:blogger.com,1999:blog-3053230616754516408.post-6057429232097996992024-01-16T10:00:00.001-05:002024-01-16T10:00:00.132-05:00Azure @ Enterprise - Postman to simulate api to api on behalf of flow<p>Often we encounter situation where we have one web api that needs to call another api as the incoming user. Basically our web api needs to impersonate the user to the next downstream api. Below goes the official diagram¹ from Microsoft.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://learn.microsoft.com/en-us/entra/identity-platform/media/v2-oauth2-on-behalf-of-flow/protocols-oauth-on-behalf-of-flow.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="295" data-original-width="595" height="317" src="https://learn.microsoft.com/en-us/entra/identity-platform/media/v2-oauth2-on-behalf-of-flow/protocols-oauth-on-behalf-of-flow.png" width="640" /></a></div>Let us call the Web API A as middle tier API and Web API B as down stream api. Here we assume our application is an Angular web app.<p></p><p>Coding that auth flow is little complex. Before coding it is always good to understand how this can be done using Postman.</p><p>This require setting up Azure app registrations which are tricky to many developers. So we will detail those out. A glance of sample setup is below</p><p></p><ul style="text-align: left;"><li>Application</li><ul><li>App registration (aad-testapp/b34ade37-*)</li><ul><li>This app registration should have Postman's redirect url "https://oauth.pstmn.io/v1/callback"</li><li>This needs to have secret to be used from Postman</li><li>This should have permission to call aad-testapi1 with admin consent to avoid runtime individual user consent.</li></ul></ul><li>Middle tier web api (Web API A)</li><ul><li>App registration (aad-testapi1/b6d5852b-*)</li><ul><li>This should have secret to be used while asking for ob behalf of token</li><li>This should have permission to call aad-testapi2 with admin consernt to avoid runtime user consent.</li></ul></ul><li>Downstream api (Web API B)</li><ul><li>App registration - (aad-testapi2/67d35db8-*)</li><ul><li>No need to have certificates or secrets as this is not getting any token.</li><li>The expose and API section to have the aad-testapi1 to authorize and to avoid any consent</li></ul></ul></ul><div>Once we have the above mental model, it would be easy to understand the relations. Below goes the screenshots for the above.</div><h2 style="text-align: left;">App registration for application (aad-testapp)</h2><div class="separator" style="clear: both; text-align: center;"><a href="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2024/01/16-postman-on-behalf-of-flow/01-aad-testapp-01-authentication.png?raw=true" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="277" data-original-width="800" height="222" src="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2024/01/16-postman-on-behalf-of-flow/01-aad-testapp-01-authentication.png?raw=true" width="640" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2024/01/16-postman-on-behalf-of-flow/02-aad-testapp-02-secrets.png?raw=true" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="287" data-original-width="800" height="230" src="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2024/01/16-postman-on-behalf-of-flow/02-aad-testapp-02-secrets.png?raw=true" width="640" /></a></div><br /><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2024/01/16-postman-on-behalf-of-flow/03-aad-testapp-api-permissions.png?raw=true" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="260" data-original-width="800" height="208" src="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2024/01/16-postman-on-behalf-of-flow/03-aad-testapp-api-permissions.png?raw=true" width="640" /></a></div><div><br /></div><h2 style="text-align: left;">App registration for Middle tier api(aad-webapi1)</h2><div class="separator" style="clear: both; text-align: center;"><a href="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2024/01/16-postman-on-behalf-of-flow/05-aad-testapi1-api-permissions.png?raw=true" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="258" data-original-width="800" height="206" src="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2024/01/16-postman-on-behalf-of-flow/05-aad-testapi1-api-permissions.png?raw=true" width="640" /></a></div><h2 style="text-align: left;"><div class="separator" style="clear: both; text-align: center;"><a href="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2024/01/16-postman-on-behalf-of-flow/05-aad-testapi1-secrets.png?raw=true" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="292" data-original-width="800" height="234" src="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2024/01/16-postman-on-behalf-of-flow/05-aad-testapi1-secrets.png?raw=true" width="640" /></a></div>App registration for down stream api(aad-webapi2)</h2><div class="separator" style="clear: both; text-align: center;"><a href="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2024/01/16-postman-on-behalf-of-flow/07-aad-testapi2-expose-api.png?raw=true" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="417" data-original-width="800" height="334" src="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2024/01/16-postman-on-behalf-of-flow/07-aad-testapi2-expose-api.png?raw=true" width="640" /></a></div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">Now let us get into business of getting tokens</div><h2 style="text-align: left;">Getting First token A</h2><div>This token is to talk between the client app and the middle tier API. This cane be done as normal way² in Postman.</div><div>We can use any url as we are not really making call to this web api. This token is used to obtain the ob behalf of token so that we can call the downstream api.</div><div><br /></div><div>Set the Authorization to OAuth 2.0.<div class="separator" style="clear: both; text-align: center;"><a href="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2024/01/16-postman-on-behalf-of-flow/12-jwt1-01-type.png?raw=true" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="268" data-original-width="362" height="237" src="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2024/01/16-postman-on-behalf-of-flow/12-jwt1-01-type.png?raw=true" width="320" /></a></div>Now in the configure new token give the details as follows.<div class="separator" style="clear: both; text-align: center;"><a href="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2024/01/16-postman-on-behalf-of-flow/13-jwt1-01-config-token.png?raw=true" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="610" data-original-width="683" height="572" src="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2024/01/16-postman-on-behalf-of-flow/13-jwt1-01-config-token.png?raw=true" width="640" /></a></div></div><p>Here below 3 fields may attract queries. </p><div class="separator" style="clear: both; text-align: center;"><a href="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2024/01/16-postman-on-behalf-of-flow/15-jwt1-01-config-token-details.png?raw=true" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="181" data-original-width="660" height="181" src="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2024/01/16-postman-on-behalf-of-flow/15-jwt1-01-config-token-details.png?raw=true" width="660" /></a></div><p></p><p>Let us details it, if the screenshot is not enough.</p><p></p><ul style="text-align: left;"><li>Auth Url - The url that ends with /authorize.</li><li>Access Token Url - The url that ends with /token.</li></ul><p></p><p>These 2 urls can be seen if we take fiddler traces when the token is requested from Postman.</p><p></p><ul style="text-align: left;"><li>Scope - the scope to the aad-webapi1.</li></ul><div>Once the values are set, click on the <i>'Get New Access Token'</i> button. This will open a browser and ask for login. Once it is success it will redirect to the <a href="https://oauth.pstmn.io/v1/callback">https://oauth.pstmn.io/v1/callback</a> which will show a popup in browser to open Postman again. If all that is success Postman will show the new token.</div><div><br /></div><div>This token we will be using as assertion in next step.</div><h2 style="text-align: left;">Getting on behalf of token B</h2><div>Now we simulated the Angular client getting the token and it will be sending that in Authorization header. Let us simulate how the middle tier API going to get the on behalf of token using this token.<div class="separator" style="clear: both; text-align: center;"><a href="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2024/01/16-postman-on-behalf-of-flow/17-jwt-02-GetAccessTokenOBO.png?raw=true" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="329" data-original-width="800" height="263" src="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2024/01/16-postman-on-behalf-of-flow/17-jwt-02-GetAccessTokenOBO.png?raw=true" width="640" /></a></div><br />The url we are using is the normal token url of the tenant.</div><div><ul style="text-align: left;"><li><i>grant_type</i> - no idea why it needs to be this string. It is as per the docs³. We normally expect this to be 'on_behalf_of', but it is not.</li><li><i>requested_token_use</i> - here comes the word 'on_behalf_of'</li><li><i>client_id</i> - is the id of middle tier app registration</li><li><i>client_secret</i> - created in middle tier app registration</li><li><i>scope</i> - poining to downstream api</li><li><i>assertion</i> - this is the incoming token to middle tier api from the client application.</li></ul><div><blockquote>When we are coding in .Net we can use X509 certificate instead of <i>client_secret</i></blockquote><i></i></div><div>Click 'Send' button to get new token. This token can be used to call the downstream api.</div></div><p></p><h2 style="text-align: left;">Reference</h2><p></p><ul style="text-align: left;"><li><a href="https://learn.microsoft.com/en-us/entra/identity-platform/v2-oauth2-on-behalf-of-flow#protocol-diagram">¹ - https://learn.microsoft.com/en-us/entra/identity-platform/v2-oauth2-on-behalf-of-flow#protocol-diagram</a></li><li>² - <a href="https://teamfolio.co.uk/blog/configure-postman-to-use-azure-active-directory-authorisation">https://teamfolio.co.uk/blog/configure-postman-to-use-azure-active-directory-authorisation</a></li><li><a href="https://learn.microsoft.com/en-us/entra/identity-platform/v2-oauth2-on-behalf-of-flow#first-case-access-token-request-with-a-shared-secret">³ - https://learn.microsoft.com/en-us/entra/identity-platform/v2-oauth2-on-behalf-of-flow#first-case-access-token-request-with-a-shared-secret</a></li></ul><p></p><br />Joymonhttp://www.blogger.com/profile/12014569049635491702noreply@blogger.com0tag:blogger.com,1999:blog-3053230616754516408.post-5948838084643832712024-01-09T10:00:00.006-05:002024-01-09T10:00:00.139-05:00Samba from Raspberry Pi - Rejected an insecure guest logon [Solved]<div style="text-align: left;">This post is about troubleshooting connectivity issues to Raspberry Pi when accessing an SMB shared path.</div><h2 style="text-align: left;">Context</h2><div>The aim is to expose a NAS from Raspberry Pi using Samba. It should be accessible only with authentication. The good thing in Windows is that it disables the anonymous access by default<span face="sans-serif" style="background-color: #f8f9fa; color: #202122; font-size: 14px;">¹</span>.</div><div>For authenticating there is a user created in Raspberry Pi. Below is the design.<div class="separator" style="clear: both; text-align: center;"><a href="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2024/01/09-rejected-insecure-login-Raspberry-Pi/01-design.png?raw=true" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="433" data-original-width="800" height="346" src="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2024/01/09-rejected-insecure-login-Raspberry-Pi/01-design.png?raw=true" width="640" /></a></div></div><h2 style="text-align: left;">Problem</h2><div>When accessing SMB share from Windows 10 machine, it error out with</div><div><br /></div><div><div class="separator" style="clear: both; font-size: 13px; text-align: center; white-space-collapse: preserve;"><a href="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2024/01/09-rejected-insecure-login-Raspberry-Pi/03-error-dialog.png?raw=true" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="178" data-original-width="480" height="238" src="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2024/01/09-rejected-insecure-login-Raspberry-Pi/03-error-dialog.png?raw=true" width="640" /></a></div>"<i>\\raspberrypi.lan is not accessible. You might not have permission to use this network resource. Contact the administrator of this server to find out if you have access permissions.</i></div><div><i><br /></i></div><div><i>You can't access this shared folder because your organization's security policies block unauthenticated guest access. These policies help protect your PC from unsafe or malicious devices on the network</i>."</div><div><br /></div><p>The event viewer shows the below.</p><p><i>Rejected an insecure guest logon.</i></p><p><i>User name: <user name></i></p><p><i>Server name: \raspberrypi</i></p><p><i>Guidance:</i></p><p><i>This event indicates that the server attempted to log the user on as an unauthenticated guest and was denied by the client. Guest logons do not support standard security features such as signing and encryption. As a result, guest logons are vulnerable to man-in-the-middle attacks that can expose sensitive data on the network. Windows disables insecure guest logons by default. Microsoft does not recommend enabling insecure guest logons.</i></p><div><span face="ui-monospace, "Cascadia Mono", "Segoe UI Mono", "Liberation Mono", Menlo, Monaco, Consolas, monospace" style="background-color: #e3e6e8; color: #0c0d0e;"><div class="separator" style="clear: both; text-align: center;"><a href="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2024/01/09-rejected-insecure-login-Raspberry-Pi/05-event-viewer.png?raw=true" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="437" data-original-width="1451" height="192" src="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2024/01/09-rejected-insecure-login-Raspberry-Pi/05-event-viewer.png?raw=true" width="640" /></a></div><h2 style="text-align: left;">Analysis</h2><div>It says the attempt is made as an unauthenticated user. But the dialog box for login was present and entered the username and password.</div><h2 style="text-align: left;">Root cause</h2><div>The credentials were wrong. When the password was wrong, instead of telling wrong user name or password, it fall back to unauthenticated anonymous user access and that is rejected.</div><h3 style="text-align: left;">Why it takes that fall back?</h3><div>It is bacause of a setting in Samba. The setting name is "map to guest". The default value is 'Never'² but the value was somehow different.</div><h2 style="text-align: left;">Solution / Fix</h2><div>Make sure the smb.conf has "map to guest user" = 'never', if we want to enforce only authenticated users accessing the share. May be we can leave it as the default value is 'never'.</div><div><br /></div><div>More more details refer Ansible scripts<span face="sans-serif" style="background-color: #f8f9fa; color: #202122; font-size: 14px;">³</span> in previous post in this blog.</div><div><br /></div><div><span style="color: black;">Happy debugging...</span></div></span></div><h2 style="text-align: left;">References</h2><div>¹ - <a href="https://learn.microsoft.com/en-us/troubleshoot/windows-server/networking/guest-access-in-smb2-is-disabled-by-default">https://learn.microsoft.com/en-us/troubleshoot/windows-server/networking/guest-access-in-smb2-is-disabled-by-default</a></div><div>² - <a href="https://www.samba.org/samba/docs/current/man-html/smb.conf.5.html#idm6514">https://www.samba.org/samba/docs/current/man-html/smb.conf.5.html#idm6514</a></div><div><span face="sans-serif" style="background-color: #f8f9fa; color: #202122; font-size: 14px;">³ - </span><a href="https://github.com/joymon/ansible-raspberry-home-nas-plex/blob/main/samba-playbook.yml#L21">https://github.com/joymon/ansible-raspberry-home-nas-plex/blob/main/samba-playbook.yml#L21</a></div>Joymonhttp://www.blogger.com/profile/12014569049635491702noreply@blogger.com0tag:blogger.com,1999:blog-3053230616754516408.post-68614943656201578042024-01-02T21:30:00.002-05:002024-01-02T21:30:00.241-05:00Review of 2023 & plans for 2024 as a software engineer<p>A review of <a href="https://joymonscode.blogspot.com/2023/01/review-of-2022-plans-for-2023-as.html" rel="nofollow" target="_blank">last year</a> and plan for 2024 except anything from the day job.</p><h2>2023 - What was planned with status</h2><h3>Goals</h3><div><ul style="text-align: left;"><li>Continue</li><ul><li style="margin: 0px 0px 0.25em; padding: 0px;"><span style="background-color: white; color: #202124; font-size: 16px;">✅</span>Python - Continue learning.</li><ul><li style="margin: 0px 0px 0.25em; padding: 0px;">Converted my own utilities to Python. Now my first preference coming to mind is Python.</li></ul><li style="margin: 0px 0px 0.25em; padding: 0px;"><span style="background-color: white; color: #202124; font-size: 14px;">❌</span>WebAssembly - Continue learning including WASM containers</li><li style="margin: 0px 0px 0.25em; padding: 0px;"><span style="background-color: white; color: #202124; font-size: 14px;">❌</span>Pass the AZ-304 exam. Continue learning Azure computing.</li><li style="margin: 0px 0px 0.25em; padding: 0px;"><span style="background-color: white; color: #202124; font-size: 16px;">✅</span>Continue making 1 blog post per week.</li><li style="margin: 0px 0px 0.25em; padding: 0px;"><span style="background-color: white; color: #202124; font-size: 14px;">❌</span>At least 25% or more posts to be videos</li><li style="margin: 0px 0px 0.25em; padding: 0px;"><span style="background-color: white; color: #202124; font-size: 14px;">❌</span>Achieve 250 subscribers in my coding YouTube channel <a href="https://www.youtube.com/channel/UC78wYrq_keVaDV8STReHRxg" style="color: #2288bb; text-decoration-line: none;" target="_blank">Joymon v/s Code</a>. </li><ul><li style="margin: 0px 0px 0.25em; padding: 0px;">Just crossed 100 only.</li></ul><li style="margin: 0px 0px 0.25em; padding: 0px;"><span style="background-color: white; color: #202124; font-size: 16px;">✅</span>Monthly 1-2 hours for personal projects</li></ul><li>New</li><ul><li><span style="background-color: white; color: #202124; font-size: 16px;">✅</span>Set up the Ethereum blockchain and write one smart contract</li><li><span style="background-color: white; color: #202124; font-size: 14px;">❌</span>Spend 3 hours every week on LinkedIn Learning</li></ul><li>Experiments</li><ul><li style="margin: 0px 0px 0.25em; padding: 0px;"><span style="background-color: white; color: #202124; font-size: 14px;">❌</span>The big thing would be an attempt to present a topic in a public event conducted by user groups.</li><li style="margin: 0px 0px 0.25em; padding: 0px;"><span style="background-color: white; color: #202124; font-size: 14px;">❌</span>At least 10% posts to external sites such as CodeProject, DZone CSharpCorner, etc.</li><li style="margin: 0px 0px 0.25em; padding: 0px;"><span style="background-color: white; color: #202124; font-size: 14px;">❌</span>Write my own GitHub action.</li><li style="margin: 0px 0px 0.25em; padding: 0px;"><span style="background-color: white; color: #202124; font-size: 14px;">❌</span>Write a test Telegram/Discord bot</li><li style="margin: 0px 0px 0.25em; padding: 0px;"><span style="background-color: white; color: #202124; font-size: 14px;">❌</span>Write a test Flutter app</li></ul></ul></div><h3>What I am not going to do</h3><div><ul><li style="margin: 0px 0px 0.25em; padding: 0px;"><span style="background-color: white; color: #202124; font-size: 16px;">✅</span>Blockchain coding - expertise.</li><li style="margin: 0px 0px 0.25em; padding: 0px;"><span style="background-color: white; color: #202124; font-size: 16px;">✅</span>Go & Scala - watch some videos. Not in deep.</li><li style="margin: 0px 0px 0.25em; padding: 0px;"><span style="background-color: white; color: #202124; font-size: 16px;">✅</span>Data science algorithms.</li></ul><div><h3>Additional achievements</h3><div><ul><li>A beginner stage in new buzzword generative AI.</li><li>Hands-on experience with Ansible by automating home nas on Raspberry Pi.</li><li>Deep dive into some of the Kubernetes features and Windows containers</li><li>Deep dive into SharePoint Online, and Office Online.</li></ul><div>Overall it could have been much better. The meaningless excuse I am telling myself is that my spouse started working hence I have very minimal time for extra activities.</div><h2 style="text-align: left;">2024 - What is ahead</h2></div></div></div><h3 style="text-align: left;">Goals</h3><div><ul style="text-align: left;"><li>Continue</li><ul><li>Continue Python</li><li>WebAssembly - Learn WASM containers.</li><li style="margin: 0px 0px 0.25em; padding: 0px;">Continue making 1 blog post per week. Out of that, 15 or more posts are to be videos. </li><li style="margin: 0px 0px 0.25em; padding: 0px;">Achieve 500 subscribers in my coding YouTube channel <a href="https://www.youtube.com/channel/UC78wYrq_keVaDV8STReHRxg" style="color: #2288bb; text-decoration-line: none;" target="_blank">Joymon v/s Code</a>.</li><li style="margin: 0px 0px 0.25em; padding: 0px;">Monthly 1-2 hours for personal projects</li><li style="margin: 0px 0px 0.25em; padding: 0px;">Intermediate level in generative AI</li></ul><li>New</li><ul><li style="margin: 0px 0px 0.25em; padding: 0px;">Same as last year, a big thing would be an attempt to present a topic in a public event conducted by user groups.</li><li style="margin: 0px 0px 0.25em; padding: 0px;">At least 10% posts to external sites such as CodeProject, DZone CSharpCorner, etc.</li><li style="margin: 0px 0px 0.25em; padding: 0px;">Write my own GitHub action.</li><li style="margin: 0px 0px 0.25em; padding: 0px;">Write a test Telegram/Discord bot.</li><li style="margin: 0px 0px 0.25em; padding: 0px;">Write a test Flutter app.</li><li>Teaching my kids how to think like programmers. Python for the 4th grader and Scratch for kindergartner.</li></ul><li>Experiments</li><ul><li>Automate YouTube video production via AI tools.</li><li>A blockchain app prototype.</li><li>Try to do a Scratch or similar technology video with my kids.</li><li>Experiment with the 3D printer at our public library and printing technology overall.</li></ul></ul><h3 style="text-align: left;">What I am not going to do</h3></div><div><ul><li style="margin: 0px 0px 0.25em; padding: 0px;">Blockchain coding - expertise.</li><li style="margin: 0px 0px 0.25em; padding: 0px;">Go & Scala - watch some videos. Not in deep.</li><li style="margin: 0px 0px 0.25em; padding: 0px;">Data science algorithms.</li></ul></div>Joymonhttp://www.blogger.com/profile/12014569049635491702noreply@blogger.com0tag:blogger.com,1999:blog-3053230616754516408.post-42994416242118095212023-12-26T22:30:00.002-05:002024-03-15T11:35:35.335-04:00Azure @ Enterprise - Entra - On behalf of flow - AADSTS50013: Assertion failed signature validation<p>Recently we were trying to achieve the below authentication scenario and we stuck with an exception as below. This post is about debugging the same issue.</p><p><span style="color: red; font-size: x-small;">"Exception MsalUiRequierdException with below message</span></p><p><span style="color: red; font-size: x-small;">"OnBehalfOfCredential authentication failed: AADSTS50013: Assertion failed signature validation. [Reason - Key was found, but use of the key to verify the signature failed., Thumbprint of key used by client: 'E41DE<cert thumbprint>D91', Found key 'Start=12/05/2023 17:16:57, End=12/05/2028 17:16:57', Please visit the Azure Portal, Graph Explorer or directly use MS Graph to see configured keys for app Id '00000000-0000-0000-0000-000000000000'. Review the documentation at https://docs.microsoft.com/en-us/graph/deployments to determine the corresponding service endpoint and https://docs.microsoft.com/en-us/graph/api/application-get?view=graph-rest-1.0&tabs=http to build a query request URL, such as 'https://graph.microsoft.com/beta/applications/00000000-0000-0000-0000-000000000000']. Trace ID: <guid> Correlation ID: <guid> Timestamp: 2023-12-22 16:15:05Z"</span></p><p><span>Below is the call stack</span></p><p><span style="color: red;"><span style="font-size: xx-small;">Azure.Identity.AuthenticationFailedException:
<br /> at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow (Azure.Identity, Version=1.10.4.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
<br /> at Azure.Identity.OnBehalfOfCredential+<gettokeninternalasync>d__19.MoveNext (Azure.Identity, Version=1.10.4.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
<br /> at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
<br /> at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
<br /> at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
<br /> at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
<br /> at System.Threading.Tasks.ValueTask`1.get_Result (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
<br /> at System.Runtime.CompilerServices.ValueTaskAwaiter`1.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
<br /> at MyAuthApi1.Controllers.WeatherForecastController+<getjwtonbehalfofuser>d__11.MoveNext (MyAuthApi1, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: /home/runner/work/aad-auth-test/aad-auth-test/MyAuthApi1/Controllers/WeatherForecastController.cs:137)<br /></getjwtonbehalfofuser></gettokeninternalasync></span></span></p><p><span style="color: red;"><span style="font-size: xx-small;"><gettokeninternalasync><getjwtonbehalfofuser>Inner exception Microsoft.Identity.Client.MsalUiRequiredException handled </getjwtonbehalfofuser></gettokeninternalasync></span></span></p><p><span style="color: red;"><span style="font-size: xx-small;"><gettokeninternalasync><getjwtonbehalfofuser>at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow:
<br /> at Microsoft.Identity.Client.Internal.Requests.RequestBase+<handletokenrefresherrorasync>d__31.MoveNext (Microsoft.Identity.Client, Version=4.57.0.0, Culture=neutral, PublicKeyToken=0a613f4dd989e8ae)
<br /> at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
<br /> at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
<br /> at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
<br /> at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
<br /> at Microsoft.Identity.Client.Internal.Requests.OnBehalfOfRequest+<executeasync>d__3.MoveNext (Microsoft.Identity.Client, Version=4.57.0.0, Culture=neutral, PublicKeyToken=0a613f4dd989e8ae)
<br /> at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
<br /> at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
<br /> at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
<br /> at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
<br /> at Microsoft.Identity.Client.Internal.Requests.RequestBase+<runasync>d__12.MoveNext (Microsoft.Identity.Client, Version=4.57.0.0, Culture=neutral, PublicKeyToken=0a613f4dd989e8ae)
<br /> at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
<br /> at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
<br /> at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
<br /> at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
<br /> at Microsoft.Identity.Client.ApiConfig.Executors.ConfidentialClientExecutor+<executeasync>d__4.MoveNext (Microsoft.Identity.Client, Version=4.57.0.0, Culture=neutral, PublicKeyToken=0a613f4dd989e8ae)
<br /> at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
<br /> at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
<br /> at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
<br /> at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
<br /> at Azure.Identity.AbstractAcquireTokenParameterBuilderExtensions+<executeasync>d__0`1.MoveNext (Azure.Identity, Version=1.10.4.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
<br /> at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
<br /> at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
<br /> at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
<br /> at System.Threading.Tasks.ValueTask`1.get_Result (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
<br /> at Azure.Identity.MsalConfidentialClient+<acquiretokenonbehalfofcoreasync>d__27.MoveNext (Azure.Identity, Version=1.10.4.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
<br /> at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
<br /> at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
<br /> at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
<br /> at System.Threading.Tasks.ValueTask`1.get_Result (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
<br /> at Azure.Identity.MsalConfidentialClient+<acquiretokenonbehalfofasync>d__26.MoveNext (Azure.Identity, Version=1.10.4.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
<br /> at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
<br /> at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
<br /> at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
<br /> at System.Threading.Tasks.ValueTask`1.get_Result (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
<br /> at Azure.Identity.OnBehalfOfCredential+<gettokeninternalasync>d__19.MoveNext (Azure.Identity, Version=1.10.4.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
</gettokeninternalasync></acquiretokenonbehalfofasync></acquiretokenonbehalfofcoreasync></executeasync></executeasync></runasync></executeasync></handletokenrefresherrorasync></getjwtonbehalfofuser></gettokeninternalasync></span></span></p><p>The message emitted has not much relation with the actual root cause. Debugging steps are later converted to the <a href="https://www.youtube.com/watch?v=jcf0_nbcZyM" rel="nofollow" target="_blank">scientific debugging</a><span style="background-color: #f8f9fa; color: #202122; font-family: sans-serif; font-size: 14px;">¹</span><a href="https://joymonscode.blogspot.com/2020/02/scientific-approach-to-software.html" rel="nofollow" target="_blank">template</a><span style="background-color: #f8f9fa; color: #202122; font-family: sans-serif; font-size: 14px;">²</span>for easier understanding. Below is the embedding of the same. Please click <a href="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/12/26-obo-aadsts50013/scientific-debugging-on_behalf_of_flow-aadsts50013_Assertion%20failed%20signature%20validation.md" rel="nofollow" target="_blank">here</a> in case it is not loading properly.</p><script src="https://emgithub.com/embed-v2.js?target=https%3A%2F%2Fgithub.com%2Fblogfiles%2Fblogfiles.github.io%2Fblob%2Fmaster%2Fjoymonscode%2F2023%2F12%2F26-obo-aadsts50013%2Fscientific-debugging-on_behalf_of_flow-aadsts50013_Assertion%2520failed%2520signature%2520validation.md&style=default&type=markdown&showBorder=on&showLineNumbers=on&showFileMeta=on&showFullPath=on&showCopy=on"></script><p>Happy debugging...</p><h2 style="text-align: left;">References</h2><div><ul style="text-align: left;"><li><span style="background-color: #f8f9fa; color: #202122; font-family: sans-serif; font-size: 14px;">¹ - </span><span style="color: #202122; font-family: sans-serif;"><span style="font-size: 14px;"><a href="https://www.youtube.com/watch?v=jcf0_nbcZyM">https://www.youtube.com/watch?v=jcf0_nbcZyM</a></span></span></li><li><span style="background-color: #f8f9fa; color: #202122; font-family: sans-serif; font-size: 14px;">²</span>- <a href="https://joymonscode.blogspot.com/2020/02/scientific-approach-to-software.html">https://joymonscode.blogspot.com/2020/02/scientific-approach-to-software.html</a></li></ul></div><div><br /></div><p><br /></p>Joymonhttp://www.blogger.com/profile/12014569049635491702noreply@blogger.com0tag:blogger.com,1999:blog-3053230616754516408.post-81694888496369331652023-12-12T12:00:00.001-05:002023-12-12T12:00:00.146-05:00Setting up home NAS - Part 7 - Automate using Ansible<div style="text-align: justify;">This is the continuation of the "Setting up home NAS" series. Better read the below posts to get context.</div><div><div dir="ltr" trbidi="on"><a href="https://joymonscode.blogspot.com/2020/05/setting-up-home-nas-part-0-raid-vs.html" target="_blank">Setting up home NAS - Part 0 - RAID v/s Backup rsync v/s Raspberry Pi v/s Cloud Storage for Youtuber in the year 2020</a></div><div dir="ltr" trbidi="on"><a href="https://joymonscode.blogspot.com/2020/04/setting-up-home-nas-part-1-scientific.html">Setting up home NAS - Part 1 - Scientific debugging to get USB drive visible as \\share in network</a></div><div dir="ltr" trbidi="on"><a href="https://joymonscode.blogspot.com/2020/05/setting-up-home-nas-part-2-powershell.html" target="_blank">Setting up home NAS - Part 2 - PowerShell arrange media date bases files into YYYY\MM\DD folders</a></div><div dir="ltr" trbidi="on"><a href="https://joymonscode.blogspot.com/2020/05/setting-up-home-nas-part-3-dealing-with.html" rel="nofollow" target="_blank">Setting up home NAS - Part 3 - Dealing with duplicates</a></div><div dir="ltr" trbidi="on"><a href="https://joymonscode.blogspot.com/2020/05/setting-up-home-nas-part-5-raspberry-pi.html" rel="nofollow" target="_blank">Setting up home NAS - Patt 4 - Raspberry Pi to build locally redundant NAS for home</a></div><div dir="ltr" trbidi="on"><a href="https://joymonscode.blogspot.com/2022/03/setting-up-home-nas-part-5-e-mailing.html" rel="nofollow" target="_blank">Setting up home NAS - Part 5 - e-mailing rsync report</a></div></div><div><a href="https://joymonscode.blogspot.com/2023/12/setting-up-home-nas-part-6-raspberry-pi.html" rel="nofollow" target="_blank">Setting up home NAS - Part 6 - Raspberry Pi as Media Server using Plex</a></div><div>Setting up home NAS - Part 7 - Automate using Ansible</div><h2 style="text-align: left;">Introduction</h2><div style="text-align: justify;">Previous posts have manual instructions to set up the Raspberry Pi to function as a NAS and Plex media server. As we all know manual steps are error-prone and difficult to replicate. This post is about automating the deployment and manual configuration steps.</div><h2 style="text-align: left;">Architecture</h2><div>There are different approaches to automating the machine configuration and software deployment.</div><div><br /></div><div>The easy way is to simply write scripts and get them executed in the target machine itself. Here target is Raspberry Pi and Ansible supports this method and lets us install it on Raspberry Pi.</div><div><br /></div><div>Secondly, agent-based<span style="background-color: #f8f9fa; color: #202122; font-family: sans-serif; font-size: 14px;">¹</span> where we first manually install an agent software that polls the central repository of configuration. In case there are any changes, it will apply those to the host machine where it is running.</div><div><br /></div><div>Another way is the opposite of agent-based. A controller machine pushes the changes to the target machine using some remoting technology. Here the target is Raspberry Pi</div><div class="separator" style="clear: both; text-align: center;"><a href="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/12/12-nas-ansible/01-ssh-raspi.png?raw=true" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="300" data-original-width="800" height="240" src="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/12/12-nas-ansible/01-ssh-raspi.png?raw=true" width="640" /></a></div><br /><div><br /></div><div>Here we are using an agentless method using a tool called Red Hat Ansible². Ansible is a well-known tool in IT automation workflows.</div><h2 style="text-align: left;">SSH to Pi with keys </h2><p style="text-align: justify;">Since there is no agent to listen in, the controller machine needs to use a way to remote into the target machine to be configured. Ansible uses SSH³ to do the same.</p><p style="text-align: justify;">The default way is to run the Ansible scripts and it will ask for login details. For Raspberry to set up that is fine. But there are ways to avoid it by using the SSH public-private key pairs.</p><p>There are clearly documented ways to establish a passwordless login to RasPi⁴ using SSH keys. But there are chances of things going wrong and troubleshooting SSH keys to authenticate should be a separate post on its own.</p><h2 style="text-align: left;">Installing Ansible</h2><div>It requires Python to be installed. The installation of Ansible is straightforward and well documented⁵.</div><h2 style="text-align: left;">Source code</h2><p>Enough theory. Since it involves scripts that can be enhanced later, it is better to keep in a source control repository. Below is the source code repo link to GitHub.</p><p></p><blockquote><a href="https://github.com/joymon/ansible-raspberry-home-nas-plex">https://github.com/joymon/ansible-raspberry-home-nas-plex</a></blockquote><p></p><p>The readme.md is pretty much updated.</p><p></p><blockquote>One thing pending is the setting up of the user password that is used to access the shared path from Raspberry.</blockquote><p>It will be added soon and updates will be pushed to the same repo. </p><p></p><h2 style="text-align: left;">Ansible in Windows v/s Linux</h2><div>One drawback is that Ansible is not very compatible with Windows as a controller. So we have to use the WSL2 feature of Windows to manage the target Raspberry Pi.</div><div><h2 style="text-align: left;">Reference</h2></div><p></p><ul style="text-align: left;"><li><span style="background-color: #f8f9fa; color: #202122; font-family: sans-serif; font-size: 14px;">¹ - </span><a href="https://dzone.com/articles/part-ii-choosing-the-right-infrastructure-as-code">https://dzone.com/articles/part-ii-choosing-the-right-infrastructure-as-code</a></li><li>² - <a href="https://www.ansible.com/">https://www.ansible.com/</a></li><li><a href="https://docs.ansible.com/ansible/latest/inventory_guide/connection_details.html">³ - https://docs.ansible.com/ansible/latest/inventory_guide/connection_details.html</a></li><li>⁴ - <a href="https://www.raspberrypi.com/documentation/computers/remote-access.html#secure-shell-from-windows-10">https://www.raspberrypi.com/documentation/computers/remote-access.html#secure-shell-from-windows-10</a></li><li>⁵ - <a href="https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html">https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html</a></li><li>⁶ - <a href="https://docs.ansible.com/ansible/latest/os_guide/windows_faq.html#can-ansible-run-on-windows">https://docs.ansible.com/ansible/latest/os_guide/windows_faq.html#can-ansible-run-on-windows</a></li></ul><p></p>Joymonhttp://www.blogger.com/profile/12014569049635491702noreply@blogger.com0tag:blogger.com,1999:blog-3053230616754516408.post-84359626115612909742023-12-05T14:00:00.001-05:002023-12-06T12:30:50.148-05:00Setting up home NAS - Part 6 - Raspberry Pi as Plex Media Server<div>This is the continuation of the "Setting up home NAS" series. Better read the below posts to get context.</div><div><div dir="ltr" trbidi="on"><a href="https://joymonscode.blogspot.com/2020/05/setting-up-home-nas-part-0-raid-vs.html" target="_blank">Setting up home NAS - Part 0 - RAID v/s Backup rsync v/s Raspberry Pi v/s Cloud Storage for Youtuber in the year 2020</a></div><div dir="ltr" trbidi="on"><a href="https://joymonscode.blogspot.com/2020/04/setting-up-home-nas-part-1-scientific.html">Setting up home NAS - Part 1 - Scientific debugging to get USB drive visible as \\share in network</a></div><div dir="ltr" trbidi="on"><a href="https://joymonscode.blogspot.com/2020/05/setting-up-home-nas-part-2-powershell.html" target="_blank">Setting up home NAS - Part 2 - PowerShell arrange media date bases files into YYYY\MM\DD folders</a></div><div dir="ltr" trbidi="on"><a href="https://joymonscode.blogspot.com/2020/05/setting-up-home-nas-part-3-dealing-with.html" rel="nofollow" target="_blank">Setting up home NAS - Part 3 - Dealing with duplicates</a></div><div dir="ltr" trbidi="on"><a href="https://joymonscode.blogspot.com/2020/05/setting-up-home-nas-part-5-raspberry-pi.html" rel="nofollow" target="_blank">Setting up home NAS - Patt 4 - Raspberry Pi to build locally redundant NAS for home</a></div><div dir="ltr" trbidi="on"><a href="https://joymonscode.blogspot.com/2022/03/setting-up-home-nas-part-5-e-mailing.html" rel="nofollow" target="_blank">Setting up home NAS - Part 5 - e-mailing rsync report</a></div></div><div>Setting up home NAS - Part 6 - Raspberry Pi as Media Server using Plex (This post)</div><h2 style="text-align: left;">Introduction</h2><div>This post discuss how to enable Plex media server on Raspberry Pi and use within home network.</div><h2 style="text-align: left;">Approach</h2><div>At high level we have to do the below steps</div><div><ul style="text-align: left;"><li>Enable Plex package sources and install Plex</li><li>Browse to <a href="http://raspberrypi:32400/web">http://raspberrypi:32400/web</a> and associate with our Plex account</li><li>Configure the Plex to include our media folders</li></ul><div>Let's start.</div><h2 style="text-align: left;">Enable Plex package sources and install Plex</h2></div><div>Below goes the Linux commands to do the same. It needs to be run on Raspberry.</div><div><br /></div><div><div style="background-color: #1f1f1f; color: #cccccc; font-family: Consolas, "Courier New", monospace; font-size: 14px; line-height: 19px; white-space: pre;"><div><span style="color: #dcdcaa;">sudo</span> <span style="color: #ce9178;">apt-get</span> <span style="color: #ce9178;">update</span></div><div><span style="color: #dcdcaa;">sudo</span> <span style="color: #ce9178;">apt-get</span> <span style="color: #ce9178;">upgrade</span></div><div><span style="color: #6a9955;"># Enable the package install over https</span></div><div><span style="color: #dcdcaa;">sudo</span> <span style="color: #ce9178;">apt-get</span> <span style="color: #ce9178;">install</span> <span style="color: #ce9178;">apt-transport-https</span></div><div><span style="color: #6a9955;"># Adding the Plex GPG key</span></div><div><span style="color: #dcdcaa;">sudo</span> <span style="color: #ce9178;">curl</span> <span style="color: #ce9178;">https://downloads.plex.tv/plex-keys/PlexSign.key</span> <span style="color: #d4d4d4;">|</span> <span style="color: #dcdcaa;">sudo</span> <span style="color: #ce9178;">apt-key</span> <span style="color: #ce9178;">add</span> <span style="color: #ce9178;">-</span></div><div><span style="color: #6a9955;"># Add the Plex package sources to the apt. </span></div><div><span style="color: #6a9955;"># It basically creats a file with the deb ...</span></div><div><span style="color: #dcdcaa;">sudo</span> <span style="color: #ce9178;">echo</span> <span style="color: #ce9178;">deb</span> <span style="color: #ce9178;">https://downloads.plex.tv/repo/deb</span> <span style="color: #ce9178;">public</span> <span style="color: #ce9178;">main</span> <span style="color: #d4d4d4;">|</span> <span style="color: #dcdcaa;">sudo</span> <span style="color: #ce9178;">tee</span> <span style="color: #ce9178;">/etc/apt/sources.list.d/plexmediaserver.list</span></div><div><span style="color: #6a9955;"># We’re going to update our package list and install from the newly added repository.</span></div><div><span style="color: #dcdcaa;">sudo</span> <span style="color: #ce9178;">apt-get</span> <span style="color: #ce9178;">update</span></div><div><span style="color: #dcdcaa;">sudo</span> <span style="color: #ce9178;">apt-get</span> <span style="color: #ce9178;">install</span> <span style="color: #ce9178;">plexmediaserver</span></div></div></div><div style="text-align: left;"><br /></div><div style="text-align: left;">Added some comments to understand. In case of queries, please add comments. Once Plex is installed we need to configure.</div><h2 style="text-align: left;">Browse and associate with Plex Account</h2><div>First naviate to <a href="http://raspberrypi:32400/web">http://raspberrypi:32400/web</a> . The port 32400 is the default port for Plex. Please make sure the protocol prefix is <i>http</i> not <i>https</i>. It will load the screen as follows.<div class="separator" style="clear: both; text-align: center;"><a href="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/12/05-nas-plex-install/03-plex-first-screen.png?raw=true" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="527" data-original-width="800" height="422" src="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/12/05-nas-plex-install/03-plex-first-screen.png?raw=true" width="640" /></a></div><br /></div><div><br /></div><div>Note that the URL changed from our RaspberryPi host name to <a href="http://app.plex.tv">app.plex.tv</a> after some redirects. Click on sign-in and use approprate method. Once signed in, it will redirect back to the page as shows below.</div><div><br /></div><div><blockquote>At this point our local RaspberryPi hosted Plex is connected to the credential we provided.</blockquote><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/12/05-nas-plex-install/04-loggedin.png?raw=true" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="492" data-original-width="800" height="394" src="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/12/05-nas-plex-install/04-loggedin.png?raw=true" width="640" /></a></div><br /></div><div>This is simply saying how it works. Once you click on got it. It asks for the Plex pass.<div class="separator" style="clear: both; text-align: center;"><a href="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/12/05-nas-plex-install/05-skip-payment.png?raw=true" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="634" data-original-width="800" height="507" src="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/12/05-nas-plex-install/05-skip-payment.png?raw=true" width="640" /></a></div><br /></div><div><br /></div><div>It we can, better support them by buying the pass. Yes, it supports totally free mode as well. We can click on close button to skip payment. Up on closing it will take to server setup.<div class="separator" style="clear: both; text-align: center;"><a href="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/12/05-nas-plex-install/06-server-setup.png?raw=true" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="685" data-original-width="687" height="685" src="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/12/05-nas-plex-install/06-server-setup.png?raw=true" width="687" /></a></div><br /></div><div>We can see it can expose our media accissible from internet. In order to do that we need a router that is supporting UPnP or NAT-PMP feature. Also the port forwarding to 32400 <span style="background-color: #f8f9fa; color: #202122; font-family: sans-serif; font-size: 14px;">¹</span>. This tutorial is moving forward to restrct access only with the home network, not internet. In case we change our mind, it can enable later as well.</div><div><br /></div><div>The next step is to add our media folders to the Plex.<div class="separator" style="clear: both; text-align: center;"><a href="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/12/05-nas-plex-install/07-Organize-media.png?raw=true" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="601" data-original-width="652" height="590" src="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/12/05-nas-plex-install/07-Organize-media.png?raw=true" width="640" /></a></div><br /></div><div>Click on the add library <div class="separator" style="clear: both; text-align: center;"><a href="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/12/05-nas-plex-install/09-add-lib.png?raw=true" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="632" data-original-width="766" height="528" src="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/12/05-nas-plex-install/09-add-lib.png?raw=true" width="640" /></a></div><br /></div><div>This tutorial is adding an audio library of type Music.<div class="separator" style="clear: both; text-align: center;"><a href="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/12/05-nas-plex-install/10-audio-lib.png?raw=true" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="620" data-original-width="761" height="521" src="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/12/05-nas-plex-install/10-audio-lib.png?raw=true" width="640" /></a></div>The library name will be Music. Click next.<div class="separator" style="clear: both; text-align: center;"><a href="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/12/05-nas-plex-install/12-audio-add-folders.png?raw=true" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="628" data-original-width="762" height="527" src="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/12/05-nas-plex-install/12-audio-add-folders.png?raw=true" width="640" /></a></div>Click on 'Browse for Media Folder' to get the media folder selection screen.</div><div class="separator" style="clear: both; text-align: center;"><a href="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/12/05-nas-plex-install/13-browse-addfolder.png?raw=true" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="622" data-original-width="756" height="622" src="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/12/05-nas-plex-install/13-browse-addfolder.png?raw=true" width="756" /></a></div><br /><div>Browse the folders or enter the path to our folder containing audio files. This is the path in Raspberry. The path in the screenshot is where the external hard drive is mounted that contains files. Click on the "Add" button and will land on the next screen.<div class="separator" style="clear: both; text-align: center;"><a href="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/12/05-nas-plex-install/14-add-lib-button.png?raw=true" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="619" data-original-width="757" height="523" src="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/12/05-nas-plex-install/14-add-lib-button.png?raw=true" width="640" /></a></div>Click on the add library to add finish adding folder to Plex.</div><div class="separator" style="clear: both; text-align: center;"><a href="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/12/05-nas-plex-install/15-lib-added.png?raw=true" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="669" data-original-width="601" height="640" src="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/12/05-nas-plex-install/15-lib-added.png?raw=true" width="575" /></a></div><br /><div>The same task of adding folder(s) can be done after completing the initial setup also. Click on Next.<div class="separator" style="clear: both; text-align: center;"><a href="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/12/05-nas-plex-install/17-finish.png?raw=true" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="643" data-original-width="601" height="640" src="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/12/05-nas-plex-install/17-finish.png?raw=true" width="598" /></a></div><br /></div><div>At last we are done by clicking the "Done" button. It is time to navigate to the Plex that we setup. Browse to <a href="http://raspberrypi:32400/web">http://raspberrypi:32400/web</a> and we will reach the screen as below. </div><div>Again please note the Url protocol is http not the secure https.</div><div class="separator" style="clear: both; text-align: center;"><a href="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/12/05-nas-plex-install/19-home-more.png?raw=true" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="455" data-original-width="800" height="364" src="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/12/05-nas-plex-install/19-home-more.png?raw=true" width="640" /></a></div>What ever content we are seeing in our home page is free to watch from Plex. They change over time. These are not from our Raspberry Pi server. In order to see the personal contents hosted on Raspberry Pi, click the More button.<div class="separator" style="clear: both; text-align: center;"><a href="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/12/05-nas-plex-install/20-audio-library-raspberry.png?raw=true" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="499" data-original-width="800" height="399" src="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/12/05-nas-plex-install/20-audio-library-raspberry.png?raw=true" width="640" /></a></div>Here comes our personal content. Remember we created the Music library during the setup, that is what visible here.<br /><br />Now it is time to access it via the Plex web site. Naviagate to <a href="https://app.plex.tv">https://app.plex.tv</a> in the browser.<div class="separator" style="clear: both; text-align: center;"><a href="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/12/05-nas-plex-install/22-plex-not-secure.png?raw=true" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="452" data-original-width="800" height="362" src="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/12/05-nas-plex-install/22-plex-not-secure.png?raw=true" width="640" /></a></div>We can see our personal contents added to Raspberry Plex server here as well. Even the web site is <a href="http://app.plex.tv">app.plex.tv</a>, it is able to get data from Raspberry due to the connection we established between our Plex credentials and the Plex server during the setup.<div><br /><div>In case there are issues like secure connection issue as shown above, we need to wait till the DNS is propagated. Plex is using some documented magic<span style="background-color: #f8f9fa; color: #202122; font-family: sans-serif; font-size: 14px;">²</span> to retrieve the contents from Raspberry Pi to their web site and apps through our home network.</div><div><div class="separator" style="clear: both; text-align: center;"><a href="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/12/05-nas-plex-install/27-dns-magic.png?raw=true" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="386" data-original-width="692" height="357" src="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/12/05-nas-plex-install/27-dns-magic.png?raw=true" width="640" /></a></div>Since it is magic sometimes it won't work due to specifics of settings in our network such as router configuration. If all works fine, we can browse <a href="https://app.plex.tv">https://app.plex.tv</a> and view the contents. Also, it works with the other Plex app supported media players such as smart TVs, mobiles etc...<div class="separator" style="clear: both; text-align: center;"><a href="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/12/05-nas-plex-install/30-working.png?raw=true" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="415" data-original-width="800" height="332" src="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/12/05-nas-plex-install/30-working.png?raw=true" width="640" /></a></div></div><div><h2 style="text-align: left;">Configure to include our media folders</h2><div>If we need to add other folders to Plex, mouse over the name of server (here it is Raspberrypi) use the + button and point to the right media folders in Raspberry Pi.<div class="separator" style="clear: both; text-align: center;"><a href="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/12/05-nas-plex-install/31-add-folder.png?raw=true" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="532" data-original-width="800" height="426" src="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/12/05-nas-plex-install/31-add-folder.png?raw=true" width="640" /></a></div></div><h2 style="text-align: left;">Media organization</h2><div>Note that when we add the folder, Plex has to do indexing behind the scene to a database<span style="background-color: #f8f9fa; color: #202122; font-family: sans-serif; font-size: 14px;">³</span>. Depending on the storage speed and Raspberry specs, indexing time varies. After indexing only the media files will start showing up in the UI.</div><div>When we add new media files or remove, we need to run scanning<span style="background-color: #f8f9fa; color: #202122; font-family: sans-serif; font-size: 14px;">⁴</span> again to update the Plex index.</div><div><br /></div><div>If there is more interest, have a look at the below references. Have fun with Plex.</div><h2 style="text-align: left;">References</h2><div><ul style="text-align: left;"><li>¹ - <a href="https://support.plex.tv/articles/200289506-remote-access/" rel="nofollow" target="_blank">Enable remote access </a></li><li><span style="background-color: #f8f9fa; color: #202122; font-family: sans-serif; font-size: 14px;">²</span> - <a href="https://support.plex.tv/articles/206225077-how-to-use-secure-server-connections/" rel="nofollow" target="_blank">How to secure server connections</a>.</li><li><span style="background-color: #f8f9fa; color: #202122; font-family: sans-serif; font-size: 14px;">³ - </span><a href="https://support.plex.tv/articles/202915258-where-is-the-plex-media-server-data-directory-located/" rel="nofollow" target="_blank">Location of Plex database</a> </li><li>⁴ - <a href="https://support.plex.tv/articles/200289306-scanning-vs-refreshing-a-library/">https://support.plex.tv/articles/200289306-scanning-vs-refreshing-a-library/</a></li><li><a href="https://thepi.io/how-to-set-up-a-raspberry-pi-plex-server/">https://thepi.io/how-to-set-up-a-raspberry-pi-plex-server/</a></li><li><a href="https://jayden-chua.medium.com/plex-ify-your-raspberry-pi-bc4902bfa91a">https://jayden-chua.medium.com/plex-ify-your-raspberry-pi-bc4902bfa91a</a></li></ul></div></div></div>Joymonhttp://www.blogger.com/profile/12014569049635491702noreply@blogger.com0tag:blogger.com,1999:blog-3053230616754516408.post-10289847100544677062023-11-28T09:30:00.016-05:002023-11-28T09:30:00.185-05:00Azure @ Enterprise - Generate JWT using Service Principal +Certificate via MSAL.PS <p>One of the previous posts in this blog was to get JWT using the Service principal + certificate combination<span face="sans-serif" style="background-color: #f8f9fa; color: #202122; font-size: 14px;">¹.</span> The Az.Acount PowerShell module was used to achieve the task. That approach signs into Azure as the service principal and generates JWT. This post is to get the JWT without logging in.</p><h2 style="text-align: left;">MSAL.PS</h2><div>Microsoft Authentication Library is to interact with the Microsoft security system recently renamed Microsoft Entra². <div class="separator" style="clear: both; text-align: center;"><a href="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/11/28-MSAL.PS/00-azure-ad-new-name.png?raw=true" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="274" data-original-width="605" height="181" src="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/11/28-MSAL.PS/00-azure-ad-new-name.png?raw=true" width="400" /></a></div><br /></div><div>There are also client-side libraries to interact with it from different languages. MSAL.PS is the library to do the same from PowerShell. Though the MSAL.PS³ is superseded by Azure Az PowerShell SDK⁴, it is still worth giving a try.</div><div><br /></div><div>The below code shows how we can get the JWT using MSAL.PS.</div><div><script src="https://emgithub.com/embed-v2.js?target=https%3A%2F%2Fgithub.com%2Fblogfiles%2Fblogfiles.github.io%2Fblob%2Fmaster%2Fjoymonscode%2F2023%2F11%2F28-MSAL.PS%2Fget-jwt-using-msal.ps1&style=dark&type=code&showBorder=on&showLineNumbers=on&showFileMeta=on&showFullPath=on&showCopy=on"></script></div>The code is mostly straightforward but requires some basic understanding of the Azure security model, scope etc.<h3 style="text-align: left;">Making HTTP resource calls</h3><div>Once we have the JWT, it can be used to invoke HTTP calls or to execute a SQL Query.</div><div><div style="background-color: #1f1f1f; color: #cccccc; font-family: Consolas, "Courier New", monospace; font-size: 14px; line-height: 19px; white-space: pre;"><div><span style="color: #9cdcfe;">$token</span> <span style="color: #d4d4d4;">=</span> <span style="color: #dcdcaa;">Get-MsalToken</span> <span style="color: #9cdcfe;">@creds</span></div><div> </div><div><span style="color: #9cdcfe;">$reqHeaders</span> <span style="color: #d4d4d4;">=</span> <span style="color: #569cd6;">@</span>{</div><div> <span style="color: #ce9178;">'Authorization'</span> <span style="color: #d4d4d4;">=</span> <span style="color: #9cdcfe;">$token</span><span style="color: #dcdcaa;">.CreateAuthorizationHeader</span>()</div><div>}</div><br /><div><span style="color: #9cdcfe;">$requestUrl</span> <span style="color: #d4d4d4;">=</span> <span style="color: #ce9178;">"<YOUR RESOURCE URL>"</span></div><div><span style="color: #dcdcaa;">Invoke-RestMethod</span> <span style="color: #d4d4d4;">-</span>Uri <span style="color: #9cdcfe;">$requestUrl</span> <span style="color: #d4d4d4;">-</span>Headers <span style="color: #9cdcfe;">$reqHeaders</span></div><div></div></div></div><div><br /></div><div>The interesting thing is that the post published 2 years ago uses the up-to-date and officially supported method.</div><div>Happy working with legacy codebase.</div><h2 style="text-align: left;">References</h2><p><span face="sans-serif" style="background-color: #f8f9fa; color: #202122; font-size: 14px;">¹ - </span><a href="https://joymonscode.blogspot.com/2021/09/azure-enterprise-powershell-log-in-as.html">https://joymonscode.blogspot.com/2021/09/azure-enterprise-powershell-log-in-as.html</a></p><p><span face="sans-serif" style="background-color: #f8f9fa; color: #202122; font-size: 14px;">² - </span><a href="https://learn.microsoft.com/en-us/entra/fundamentals/new-name">https://learn.microsoft.com/en-us/entra/fundamentals/new-name</a> | <a href="https://devblogs.microsoft.com/identity/aad-rebrand/">https://devblogs.microsoft.com/identity/aad-rebrand/</a></p><p><span face="sans-serif" style="background-color: #f8f9fa; color: #202122; font-size: 14px;">³ - </span><a href="https://github.com/AzureAD/MSAL.PS">https://github.com/AzureAD/MSAL.PS</a></p><p>⁴ - <a href="https://learn.microsoft.com/en-us/powershell/azure/new-azureps-module-az">https://learn.microsoft.com/en-us/powershell/azure/new-azureps-module-az</a></p>Joymonhttp://www.blogger.com/profile/12014569049635491702noreply@blogger.com0tag:blogger.com,1999:blog-3053230616754516408.post-16259297106643495352023-11-21T14:00:00.001-05:002023-11-21T14:00:00.141-05:00SharePoint Online - Custom Raw Http calls using Graph SDK v5<p>The Graph SDK is a nuget library to interact with Graph API from .Net applications. Earlier this year they released<span face="sans-serif" style="background-color: #f8f9fa; color: #202122; font-size: 14px;">¹</span> Graph .Net SDK v5. That highly simplified the coding with fewer lines than v4 but has breaking changes from v4 even in the basic interactions. The good part is that the Graph SDKs simplify the HTTP operations to Graph API but in some cases, we may need to deal with Raw HTTP calls. </p><p>This post discusses such scenarios. A basic idea about Kiota and building API clients using Kiota<span style="background-color: #f8f9fa; color: #202122; font-family: sans-serif; font-size: 14px;">²</span> is helpful to easily understand the code.</p><h2 style="text-align: left;">Problem - Custom Url</h2><p>Support we are dealing with the cutting edge of Graph API, sometimes the API or property in the result may not be available in the Graph SDK as the SDK needs some time to catch up. Unfortunately, no examples can be included due to NDA. In such cases, we need to craft the URL ourselves but we don't need to deal with the authentication. How can we customize the URL in Graph SDK with the result object using the known types in Graph SDK?</p><h3 style="text-align: left;">Solution</h3><div>Below is the notebook<span style="background-color: #f8f9fa; color: #202122; font-family: sans-serif; font-size: 14px;">³</span> that has the code</div><div><script src="https://emgithub.com/embed-v2.js?target=https%3A%2F%2Fgithub.com%2Fdotnet-demos%2Fgraph-sdk-sharepoint-online-notebooks%2Fblob%2Fmain%2Fcustom-rawhttp-calls.ipynb&style=default&type=ipynb&showBorder=on&showLineNumbers=on&showFileMeta=on&showFullPath=on&showCopy=on"></script>Hope the code is self-explanatory. The catch here is the V5 SDK will serialize the result for us. Here we are getting the known <i>Drive</i> object</div><h2 style="text-align: left;">Problem - Custom URL and Raw JSON results</h2><p>Sometimes we need to deal with totally custom URLs and the result JSON. In such scenarios, we can only leverage the authentication mechanism of Graph SDK only not even the result object. How can we have a custom HTTP URL and raw JSON result using Graph SDK?</p><h3 style="text-align: left;">Solution - Using IRequestAdapter</h3><p>One way is to use the IRequestAdapter and have NativeResponseHandler. The code in a notebook<span style="background-color: #f8f9fa; color: #202122; font-family: sans-serif; font-size: 14px;">⁴</span> is below.<script src="https://emgithub.com/embed-v2.js?target=https%3A%2F%2Fgithub.com%2Fdotnet-demos%2Fgraph-sdk-sharepoint-online-notebooks%2Fblob%2Fmain%2Fcustom-rawhttp-IRequestAdapter-json.ipynb&style=default&type=ipynb&showBorder=on&showLineNumbers=on&showFileMeta=on&showFullPath=on&showCopy=on"></script> This uses a dummy object to satisfy the adapter and get the JSON via NativeResponseHander.</p><h3 style="text-align: left;">Solution - Using native HttpClient</h3><p>The above does not fully give the native experience. To get the feel of native experience we can have the below approach<span style="background-color: #f8f9fa; color: #202122; font-family: sans-serif; font-size: 14px;">⁵</span> that uses familiar HttpClient.</p><script src="https://emgithub.com/embed-v2.js?target=https%3A%2F%2Fgithub.com%2Fdotnet-demos%2Fgraph-sdk-sharepoint-online-notebooks%2Fblob%2Fmain%2Fcustom-rawhttp-HttpClient-json.ipynb&style=default&type=ipynb&showBorder=on&showLineNumbers=on&showFileMeta=on&showFullPath=on&showCopy=on"></script> Please have a look at the warning.<h2 style="text-align: left;">References</h2><p><span face="sans-serif" style="background-color: #f8f9fa; color: #202122; font-size: 14px;">¹ - </span><a href="https://devblogs.microsoft.com/microsoft365dev/microsoft-graph-net-sdk-v5-is-now-generally-available/">https://devblogs.microsoft.com/microsoft365dev/microsoft-graph-net-sdk-v5-is-now-generally-available/</a></p><p>² - <a href="https://learn.microsoft.com/en-us/openapi/kiota/quickstarts/dotnet">https://learn.microsoft.com/en-us/openapi/kiota/quickstarts/dotnet</a></p><p>³ - <a href="https://github.com/dotnet-demos/graph-sdk-sharepoint-online-notebooks/blob/main/custom-rawhttp-calls.ipynb">https://github.com/dotnet-demos/graph-sdk-sharepoint-online-notebooks/blob/main/custom-rawhttp-calls.ipynb</a></p><p>⁴ - <a href="https://github.com/dotnet-demos/graph-sdk-sharepoint-online-notebooks/blob/main/custom-rawhttp-IRequestAdapter-json.ipynb">https://github.com/dotnet-demos/graph-sdk-sharepoint-online-notebooks/blob/main/custom-rawhttp-IRequestAdapter-json.ipynb</a></p><p>⁵ - <a href="https://github.com/dotnet-demos/graph-sdk-sharepoint-online-notebooks/blob/main/custom-rawhttp-HttpClient-json.ipynb">https://github.com/dotnet-demos/graph-sdk-sharepoint-online-notebooks/blob/main/custom-rawhttp-HttpClient-json.ipynb</a></p>Joymonhttp://www.blogger.com/profile/12014569049635491702noreply@blogger.com0tag:blogger.com,1999:blog-3053230616754516408.post-30955400455873311992023-11-14T16:28:00.002-05:002023-11-14T16:28:00.143-05:00SharePoint Online - Which PnP SDK to download large files greater than 2GB<p style="text-align: justify;">As we know there are 2 PnP nuget SDKs to deal with SharePoint Online from .Net client applications. PnP.Framework and PnP.Core to name and this post discuss the approach to download files larger than 2 GB. The 2 GB is an important mark as some methods work only till the MaxValue of the integer¹ data type which is 2147483647.</p><h2 style="text-align: left;">PnP.Framework to download > 2GB files from SharePoint Online</h2><div>Below is the code to download the file and save it as a local file.</div><div><br /></div><div><div style="background-color: #1f1f1f; color: #cccccc; font-family: Consolas, "Courier New", monospace; font-size: 14px; line-height: 19px; white-space: pre;"><div><span style="color: #569cd6;">var</span> <span style="color: #9cdcfe;">file</span> <span style="color: #d4d4d4;">=</span> <span style="color: #9cdcfe;">clientContext</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">Web</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">GetFileByUrl</span>(<span style="color: #9cdcfe;">driveItemIdOrUrl</span>);</div><div><span style="color: #9cdcfe;">clientContext</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">Load</span>(<span style="color: #9cdcfe;">file</span>);</div><div><span style="color: #569cd6;">await</span> <span style="color: #9cdcfe;">clientContext</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">ExecuteQueryAsync</span>();</div><div><span style="color: #9cdcfe;">logger</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">LogInformation</span>(<span style="color: #ce9178;">$"</span><span style="color: #ce9178;">File exists in SharePoint. Size:</span><span style="color: #ce9178;">{</span><span style="color: #9cdcfe;">file</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">Length</span><span style="color: #ce9178;">:</span><span style="color: #ce9178;">0</span><span style="color: #ce9178;">,#</span><span style="color: #ce9178;">}</span><span style="color: #ce9178;">"</span>);</div><div><span style="color: #9cdcfe;">ClientResult</span><<span style="color: #9cdcfe;">Stream</span>> <span style="color: #9cdcfe;">streamFromSPO</span> <span style="color: #d4d4d4;">=</span> <span style="color: #9cdcfe;">file</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">OpenBinaryStream</span>();</div><div><span style="color: #569cd6;">await</span> <span style="color: #9cdcfe;">clientContext</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">ExecuteQueryAsync</span>();</div><br /><div><span style="color: #569cd6;">var</span> <span style="color: #9cdcfe;">localFilePathToDownload</span> <span style="color: #d4d4d4;">=</span> <span style="color: #9cdcfe;">Path</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">Combine</span>(<span style="color: #9cdcfe;">localFolderPathToDownload</span>, <span style="color: #9cdcfe;">file</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">Name</span>);</div><div><span style="color: #569cd6;">using</span> (<span style="color: #9cdcfe;">Stream</span> <span style="color: #9cdcfe;">fileStream</span> <span style="color: #d4d4d4;">=</span> <span style="color: #569cd6;">new</span> <span style="color: #9cdcfe;">FileStream</span>(<span style="color: #9cdcfe;">localFilePathToDownload</span>, <span style="color: #9cdcfe;">FileMode</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">Create</span>))</div><div>{</div><div><span style="color: #569cd6;"><span> </span>using</span> (<span style="color: #9cdcfe;">streamFromSPO</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">Value</span>)</div><div><span> </span>{</div><div> <span> </span><span style="color: #9cdcfe;">streamFromSPO</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">Value</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">CopyTo</span>(<span style="color: #9cdcfe;">fileStream</span>);</div><div> <span> </span>}</div><div>}</div><div><span style="color: #9cdcfe;">logger</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">LogInformation</span>(<span style="color: #ce9178;">$"</span><span style="color: #ce9178;">Downloaded to </span><span style="color: #ce9178;">{</span><span style="color: #9cdcfe;">localFilePathToDownload</span><span style="color: #ce9178;">}</span><span style="color: #ce9178;">"</span>);</div></div></div><div><br /></div><div>It works fine when the file size is below the max value of the integer. </div><div class="separator" style="clear: both; text-align: center;"><a href="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/11/14-spo-pnp-large-file-download/03-pnp-fwk-download-lessthan-2gb.png?raw=true" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="537" data-original-width="1042" height="330" src="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/11/14-spo-pnp-large-file-download/03-pnp-fwk-download-lessthan-2gb.png?raw=true" width="640" /></a></div><br /><div>Now let us see what happens when the file size is larger.</div><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/11/14-spo-pnp-large-file-download/05-pnp-fwk-mime-ex-4gb.png?raw=true" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="542" data-original-width="1121" height="309" src="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/11/14-spo-pnp-large-file-download/05-pnp-fwk-mime-ex-4gb.png?raw=true" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;"></td></tr></tbody></table><div><br /></div><div>System.FormatException: 'Invalid MIME content-length header encountered on read.'</div><div>There is one more method called OpenBinaryStream with options² that accepts SPOpenBinaryOption' enum. But there is nothing that helps us to get more than 2 GB files.</div><div><br /></div><div>There is another method called OpenBinaryDirect³ to download files. But it fails with 401 Unauthorized when using JWT for authentication<span style="background-color: #f8f9fa; color: #202122; font-family: sans-serif; font-size: 14px;">⁴</span>. Not even able to download smaller files.</div><div><div class="separator" style="clear: both; text-align: center;"><a href="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/11/14-spo-pnp-large-file-download/07-pnp-fwk-openbinarydirect-ex-401-4gb.png?raw=true" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="565" data-original-width="1142" height="317" src="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/11/14-spo-pnp-large-file-download/07-pnp-fwk-openbinarydirect-ex-401-4gb.png?raw=true" width="640" /></a></div>As we can see in the above image the clientContext object is the same that was used to obtain the file details.</div><div><br /></div><div>Up on Google, we can get different suggestions such as doing it all ourselves using raw HTTP calls⁵ to avoid it and completely ⁶ avoid and migrate to PnP.Core. </div><h2 style="text-align: left;">PnP.Core to download > 2GB files from SharePoint Online</h2><p>Below is the code to download a file using PnP.Core.</p><div style="background-color: #1f1f1f; color: #cccccc; font-family: Consolas, "Courier New", monospace; font-size: 14px; line-height: 19px; white-space: pre;"><div><span style="color: #569cd6;">var</span> <span style="color: #9cdcfe;">file</span> <span style="color: #d4d4d4;">=</span> <span style="color: #569cd6;">await</span> <span style="color: #9cdcfe;">pnpContext</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">Web</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">GetFileByServerRelativeUrlAsync</span>(<span style="color: #9cdcfe;">fileUrl</span>);</div><div><span style="color: #9cdcfe;">logger</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">LogInformation</span>(<span style="color: #ce9178;">$"</span><span style="color: #ce9178;">File exists in SharePoint. Size:</span>{<span style="color: #9cdcfe;">file</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">Length</span>}<span style="color: #ce9178;">"</span><span style="color: #ce9178;">)</span><span style="color: #f44747;">;</span></div><div><span style="color: #9cdcfe;">Stream</span> <span style="color: #9cdcfe;">streamFromSPO</span> <span style="color: #d4d4d4;">=</span> <span style="color: #569cd6;">await</span> <span style="color: #9cdcfe;">file</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">GetContentAsync</span>(<span style="color: #569cd6;">true</span>);</div><div><span style="color: #569cd6;">var</span> <span style="color: #9cdcfe;">localFilePathToDownload</span> <span style="color: #d4d4d4;">=</span> <span style="color: #9cdcfe;">Path</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">Combine</span>(<span style="color: #9cdcfe;">localFolderPathToDownload</span>, <span style="color: #9cdcfe;">file</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">Name</span>);</div><div><span style="color: #569cd6;">using</span> (<span style="color: #9cdcfe;">Stream</span> <span style="color: #9cdcfe;">fileStream</span> <span style="color: #d4d4d4;">=</span> <span style="color: #569cd6;">new</span> <span style="color: #9cdcfe;">FileStream</span>(<span style="color: #9cdcfe;">localFilePathToDownload</span>, <span style="color: #9cdcfe;">FileMode</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">Create</span>))</div><div>{</div><div><span style="color: #569cd6;"><span> </span>using</span> (<span style="color: #9cdcfe;">streamFromSPO</span>)</div><div> {</div><div> <span> </span><span style="color: #9cdcfe;">streamFromSPO</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">CopyTo</span>(<span style="color: #9cdcfe;">fileStream</span>);</div><div> }</div><div>}</div><div><span style="color: #9cdcfe;">logger</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">LogInformation</span>(<span style="color: #ce9178;">$"</span><span style="color: #ce9178;">{</span><span style="color: #569cd6;">nameof</span><span style="color: #ce9178;">(</span><span style="color: #dcdcaa;">DownloadFileWithReplace</span><span style="color: #ce9178;">)}</span><span style="color: #ce9178;"> - Downloaded to </span><span style="color: #ce9178;">{</span><span style="color: #9cdcfe;">localFilePathToDownload</span><span style="color: #ce9178;">}</span><span style="color: #ce9178;">"</span>);</div></div><p>It works fine for files smaller than the max value of the integer.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/11/14-spo-pnp-large-file-download/10-pnp-core-2gb-success.png?raw=true" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="425" data-original-width="1120" height="243" src="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/11/14-spo-pnp-large-file-download/10-pnp-core-2gb-success.png?raw=true" width="640" /></a></div><br /><p></p><p>What happens when the size increases above that limit?</p><div class="separator" style="clear: both; text-align: center;"><a href="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/11/14-spo-pnp-large-file-download/12-pnp-core-4gb-success.png?raw=true" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="446" data-original-width="1128" height="253" src="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/11/14-spo-pnp-large-file-download/12-pnp-core-4gb-success.png?raw=true" width="640" /></a></div><p></p><p>It works!! The Int32.MaxValue didn't cause any problem here. But there are some occasions when we may experience an issue with the stream.CopyTo method. It can even happen regardless the process is 64-bit or not. In such situations, we need to tackle them separately.</p><p>One way is to pass the Buffer Size to the stream.CopyTo method.</p><p><span style="background-color: #1f1f1f; color: #9cdcfe; font-family: Consolas, "Courier New", monospace; font-size: 14px; white-space: pre;">streamFromSPO</span><span style="background-color: #1f1f1f; color: #d4d4d4; font-family: Consolas, "Courier New", monospace; font-size: 14px; white-space: pre;">.</span><span style="background-color: #1f1f1f; color: #9cdcfe; font-family: Consolas, "Courier New", monospace; font-size: 14px; white-space: pre;">CopyTo</span><span style="background-color: #1f1f1f; color: #cccccc; font-family: Consolas, "Courier New", monospace; font-size: 14px; white-space: pre;">(</span><span style="background-color: #1f1f1f; color: #9cdcfe; font-family: Consolas, "Courier New", monospace; font-size: 14px; white-space: pre;">fileStream,BufferSize</span><span style="background-color: #1f1f1f; color: #cccccc; font-family: Consolas, "Courier New", monospace; font-size: 14px; white-space: pre;">);</span></p><p>Another way is to chunk ourselves. The code is below for our own chunking.</p><div style="background-color: #1f1f1f; color: #cccccc; font-family: Consolas, "Courier New", monospace; font-size: 14px; line-height: 19px; white-space: pre;"><div><span style="color: #569cd6;">var</span> <span style="color: #9cdcfe;">file</span> <span style="color: #d4d4d4;">=</span> <span style="color: #569cd6;">await</span> <span style="color: #9cdcfe;">pnpContext</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">Web</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">GetFileByServerRelativeUrlAsync</span>(<span style="color: #9cdcfe;">fileUrl</span>);</div><div><span style="color: #9cdcfe;">logger</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">LogInformation</span>(<span style="color: #ce9178;">$"</span><span style="color: #ce9178;">File exists in SharePoint. Size:</span>{<span style="color: #9cdcfe;">file</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">Length</span>}<span style="color: #ce9178;">"</span><span style="color: #ce9178;">)</span><span style="color: #f44747;">;</span></div><div><span style="color: #569cd6;">var</span> <span style="color: #9cdcfe;">buffer</span> <span style="color: #d4d4d4;">=</span> <span style="color: #569cd6;">new</span> <span style="color: #569cd6;">byte</span>[<span style="color: #4ec9b0;">PnPCoreConstants</span><span style="color: #d4d4d4;">.</span><span style="color: #4fc1ff;">BufferSizeToDownloadLargeFile</span>];</div><div> </div><div><span style="color: #569cd6;">var</span> <span style="color: #9cdcfe;">localFilePathToDownload</span> <span style="color: #d4d4d4;">=</span> <span style="color: #9cdcfe;">Path</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">Combine</span>(<span style="color: #9cdcfe;">localFolderPathToDownload</span>, <span style="color: #9cdcfe;">file</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">Name</span>);</div><div><span style="color: #569cd6;">using</span> (<span style="color: #9cdcfe;">Stream</span> <span style="color: #9cdcfe;">streamFromSPO</span> <span style="color: #d4d4d4;">=</span> <span style="color: #569cd6;">await</span> <span style="color: #9cdcfe;">file</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">GetContentAsync</span>(<span style="color: #569cd6;">true</span>))</div><div>{</div><div><span style="color: #569cd6;"><span> </span>using</span> (<span style="color: #9cdcfe;">Stream</span> <span style="color: #9cdcfe;">fileStream</span> <span style="color: #d4d4d4;">=</span> <span style="color: #569cd6;">new</span> <span style="color: #9cdcfe;">FileStream</span>(<span style="color: #9cdcfe;">localFilePathToDownload</span>, <span style="color: #9cdcfe;">FileMode</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">Create</span>))</div><div> {</div><div> <span> </span><span style="color: #569cd6;">int</span> <span style="color: #9cdcfe;">bytesRead</span>; <span style="color: #569cd6;">long</span> <span style="color: #9cdcfe;">totalBytesRead</span> <span style="color: #d4d4d4;">=</span> <span style="color: #b5cea8;">0</span>; <span style="color: #569cd6;">long</span> <span style="color: #9cdcfe;">streamLength</span> <span style="color: #d4d4d4;">=</span> <span style="color: #b5cea8;">0</span>;</div><div> <span style="color: #c586c0;">if</span> (<span style="color: #9cdcfe;">streamFromSPO</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">CanSeek</span>)</div><div> {</div><div> <span> </span><span style="color: #9cdcfe;">streamLength</span> <span style="color: #d4d4d4;">=</span> <span style="color: #9cdcfe;">streamFromSPO</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">Length</span>;</div><div> <span style="color: #9cdcfe;">logger</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">LogInformation</span>(<span style="color: #ce9178;">$"</span><span style="color: #ce9178;">{</span><span style="color: #569cd6;">nameof</span><span style="color: #ce9178;">(</span><span style="color: #dcdcaa;">DownloadFileWithReplace</span><span style="color: #ce9178;">)}</span><span style="color: #ce9178;"> - File Stream length:</span><span style="color: #ce9178;">{</span><span style="color: #9cdcfe;">streamLength</span><span style="color: #ce9178;">}</span><span style="color: #ce9178;">"</span>);</div><div> }</div><div> <span style="color: #c586c0;">else</span></div><div> {</div><div> <span> </span><span style="color: #9cdcfe;">streamLength</span> <span style="color: #d4d4d4;">=</span> <span style="color: #9cdcfe;">file</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">Length</span>;</div><div> }</div><div> <span style="color: #c586c0;">while</span> ((<span style="color: #9cdcfe;">bytesRead</span> <span style="color: #d4d4d4;">=</span> <span style="color: #569cd6;">await</span> <span style="color: #9cdcfe;">streamFromSPO</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">ReadAsync</span>(<span style="color: #9cdcfe;">buffer</span>, <span style="color: #b5cea8;">0</span>, <span style="color: #9cdcfe;">buffer</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">Length</span>)) <span style="color: #d4d4d4;">!=</span> <span style="color: #b5cea8;">0</span>)</div><div> {</div><div> <span> </span><span style="color: #9cdcfe;">totalBytesRead</span> <span style="color: #d4d4d4;">+=</span> <span style="color: #9cdcfe;">bytesRead</span>;</div><div> <span style="color: #9cdcfe;">fileStream</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">Write</span>(<span style="color: #9cdcfe;">buffer</span>, <span style="color: #b5cea8;">0</span>, <span style="color: #9cdcfe;">bytesRead</span>);</div><div> <span style="color: #9cdcfe;">logger</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">LogInformation</span>(<span style="color: #ce9178;">$"</span><span style="color: #ce9178;">{</span><span style="color: #569cd6;">nameof</span><span style="color: #ce9178;">(</span><span style="color: #dcdcaa;">DownloadFileWithReplace</span><span style="color: #ce9178;">)}</span><span style="color: #ce9178;"> - Progress - Read </span><span style="color: #ce9178;">{</span><span style="color: #9cdcfe;">bytesRead</span><span style="color: #ce9178;">:</span><span style="color: #ce9178;">0</span><span style="color: #ce9178;">,#</span><span style="color: #ce9178;">}</span><span style="color: #ce9178;"> and wrote.Total </span><span style="color: #ce9178;">{</span><span style="color: #9cdcfe;">totalBytesRead</span><span style="color: #ce9178;">:</span><span style="color: #ce9178;">0</span><span style="color: #ce9178;">,#</span><span style="color: #ce9178;">}</span><span style="color: #ce9178;"> / </span><span style="color: #ce9178;">{</span><span style="color: #9cdcfe;">streamLength</span><span style="color: #ce9178;">:</span><span style="color: #ce9178;">0</span><span style="color: #ce9178;">,#</span><span style="color: #ce9178;">}</span><span style="color: #ce9178;"> to </span><span style="color: #ce9178;">{</span><span style="color: #9cdcfe;">localFilePathToDownload</span><span style="color: #ce9178;">}</span><span style="color: #ce9178;">"</span>);</div><div> }</div><div> <span> </span>}</div><div>}</div><div><span style="color: #9cdcfe;">logger</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">LogInformation</span>(<span style="color: #ce9178;">$"</span><span style="color: #ce9178;">{</span><span style="color: #569cd6;">nameof</span><span style="color: #ce9178;">(</span><span style="color: #dcdcaa;">DownloadFileWithReplace</span><span style="color: #ce9178;">)}</span><span style="color: #ce9178;"> - Downloaded to </span><span style="color: #ce9178;">{</span><span style="color: #9cdcfe;">localFilePathToDownload</span><span style="color: #ce9178;">}</span><span style="color: #ce9178;">"</span>);</div></div><p>The interesting thing is that even if we provide a 2 MB buffer size, it reads 16,384 bytes only. </p><div class="separator" style="clear: both; text-align: center;"><a href="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/11/14-spo-pnp-large-file-download/14-pnp-core-4gb-chunk.png?raw=true" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="717" data-original-width="1189" height="386" src="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/11/14-spo-pnp-large-file-download/14-pnp-core-4gb-chunk.png?raw=true" width="640" /></a></div><h2 style="text-align: left;">Conclusion</h2><div>Use the PnP.Core to download large files.</div><p></p><h2 style="text-align: left;">Reference</h2><p>¹ - <a href="https://learn.microsoft.com/en-us/dotnet/api/system.int32.maxvalue?view=net-7.0">https://learn.microsoft.com/en-us/dotnet/api/system.int32.maxvalue?view=net-7.0</a></p><p>² - SPOpenBinaryOptions - <a href="https://learn.microsoft.com/en-us/previous-versions/office/developer/sharepoint-2010/bb802697(v=office.14)">https://learn.microsoft.com/en-us/previous-versions/office/developer/sharepoint-2010/bb802697(v=office.14)</a></p><p>³ - <a href="https://learn.microsoft.com/en-us/previous-versions/office/sharepoint-csom/ee537083(v=office.15)">https://learn.microsoft.com/en-us/previous-versions/office/sharepoint-csom/ee537083(v=office.15)</a></p><p><span style="background-color: #f8f9fa; color: #202122; font-family: sans-serif; font-size: 14px;">⁴</span> - <a href="https://stackoverflow.com/questions/45674435/401-unauthorized-exception-while-downloading-file-from-sharepoint">https://stackoverflow.com/questions/45674435/401-unauthorized-exception-while-downloading-file-from-sharepoint</a></p><p>⁵ - <a href="https://piyushksingh.com/2016/08/15/download-large-files-from-sharepoint-online/">https://piyushksingh.com/2016/08/15/download-large-files-from-sharepoint-online/</a></p><p>⁶ - Use PnP.Core <a href="https://github.com/pnp/powershell/pull/1239">https://github.com/pnp/powershell/pull/1239</a></p>Joymonhttp://www.blogger.com/profile/12014569049635491702noreply@blogger.com0tag:blogger.com,1999:blog-3053230616754516408.post-45111828671353166302023-11-07T11:55:00.001-05:002023-11-07T11:55:00.141-05:00Raspberry Pi - Fixing NAS (via SMB) slowness<p>This is a continuation of the Home NAS series, especially on how to increase the speed. The NAS is set up on a Raspberry Pi 4 B model. One of the older posts was<a href="https://joymonscode.blogspot.com/2023/08/raspberry-pi-why-my-nas-via-smb-is-slow.html" rel="nofollow" target="_blank"> debugging of NAS slowness</a> and it concluded that the WiFi-5 is the bottleneck. Either Raspberry needs to be connected via cable or needs to use WiFi-6. Since the WiFi-6 is not natively supported by the RasPi the only option left was to connect via wire but it requires cabling through walls which is really a high effort. The cable-based internet connection is on the main level and the data center is upstairs. The data center here refers to a small rack where RasPi,2 external hard disks, and another laptop are sitting and running without GUI.</p><h2 style="text-align: left;">Move to 5G home internet</h2><p>Fortunately or unfortunately the coaxial cable-based internet provider Xfinity increased their pricing which triggered the search for better options. Very quickly it landed on 5G home internet providers. Cheaper in price, ultra-fast connection in my area, is portable, has a WiFi-6 router built-in, etc...</p><h2 style="text-align: left;">Why T-Mobile?</h2><p>There is no affection towards them also please note that I am not affiliated with T-Mobile in any way also this post is not sponsored by them. It is cheaper and one of my friends has it and has given good feedback. Also, it has 2 ethernet ports which seems normal in any modem-router combo.</p><h2 style="text-align: left;">Speed tests</h2><p>As usual, I checked the speed using the same iperf3 tool before installing the 5G gateway. This is important as our previous test results may not be true now.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/11/07-nas-fix/00-current.png?raw=true" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="399" data-original-width="709" height="360" src="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/11/07-nas-fix/00-current.png?raw=true" width="640" /></a></div>Yes, it is still slow. This will give a max of 10 MB/s only when copying files from NAS.<p></p><h3 style="text-align: left;">New setup</h3><p>The RasPi also the client machine connected via wire and the readings are below.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/11/07-nas-fix/03-pi-wire-client-wire.png?raw=true" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="407" data-original-width="716" height="407" src="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/11/07-nas-fix/03-pi-wire-client-wire.png?raw=true" width="716" /></a></div>Wow definitely higher than expected. It is time to copy a file and check the speed.<div class="separator" style="clear: both; text-align: center;"><a href="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/11/07-nas-fix/05-nas-speed.png?raw=true" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="297" data-original-width="460" height="413" src="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/11/07-nas-fix/05-nas-speed.png?raw=true" width="640" /></a></div>Yes, it is faster. This concluded that the SMB or anything in the RasPi or OS was not the problem. The network was the problem.<p></p><h3 style="text-align: left;">The Wi-Fi 6 test</h3><p>Lastly, the Wi-Fi 6 test was also performed.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/11/07-nas-fix/07-pi-wire-client-wifi6.png?raw=true" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="399" data-original-width="712" height="359" src="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/11/07-nas-fix/07-pi-wire-client-wifi6.png?raw=true" width="640" /></a></div>It is given great performance but not to the expected level. Anyway, the Wi-Fi was kind of free. It is wise to invest in a Wi-Fi 7 router when it arrives at an affordable cost after standards are finalized.<p></p><p>The table below is for reference.</p><table border="1"><tbody><tr><td>RasPi Network</td><td>Destination Network</td><td>iperf3 reading</td></tr><tr><td>Wi-Fi 5</td><td>Wi-Fi5</td><td>80 Mbits/sec</td></tr><tr><td>Wired</td><td>Wired</td><td>771 Mbits/sec</td></tr><tr><td>Wired</td><td>Wi-Fi6</td><td>394 Mbits/sec</td></tr></tbody></table><br /><div>Bonus screenshot below that shows the internet speed test.<div class="separator" style="clear: both; text-align: center;"><a href="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/11/07-nas-fix/10-internetspeed.png?raw=true" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="550" data-original-width="1121" height="314" src="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/11/07-nas-fix/10-internetspeed.png?raw=true" width="640" /></a></div><br /><p></p></div>Joymonhttp://www.blogger.com/profile/12014569049635491702noreply@blogger.com0tag:blogger.com,1999:blog-3053230616754516408.post-8923341035303473322023-10-31T22:00:00.012-04:002023-10-31T22:00:00.143-04:00SharePoint Online - Graph API to get all files in a drive recursively even if it has more than 5000 items<h2 style="text-align: left;">Problem</h2><p>As part of the validation, we must dump all the file details inside a SharePoint drive. Some more points below</p><li>The input is a drive graph id.</li><li>It has to be recursive</li><li>There can be 10,000+ files in the drive and the export needs to be done in a reasonable amount of time.</li><li>The expected output is CSV with the below schema.</li><blockquote style="border: none; margin: 0 0 0 40px; padding: 0px;"><li style="text-align: left;">FolderName, FileGraphId, FileName</li></blockquote><p>FileGraphId is the unique id of the file that Graph API understands. It is not Guid or integer instead it uses a 34-character wide format like '01XXXXA4QGNCJMT7XXXVDKCNXXXV6LCC4E'</p><h2 style="text-align: left;">Approaches</h2><div>There are 3 APIs to interact with SharePoint Online. The first Graph has no direct method to list all the files recursively. We have to go by individual Graph API calls to the list folders and make subsequent Graph calls to get the folders and files inside recursively. It works but is very slow and high chance of getting throttled. There is a search API but that has no guarantee of returning all the files as the indexing will take some time. </div><div><br /></div><div>If we use the PnP.Core or PnP.Frameworks, there is no way to get the Graph IDs.</div><h2 style="text-align: left;">Solution</h2><div>So lastly we have to go via Graph API but get the List items (not drive) then get the driveItem of list item to retrieve the Graph Id. The code is as follows.</div><p><span style="background-color: #1f1f1f; color: #569cd6; font-family: Consolas, "Courier New", monospace; font-size: 14px; white-space: pre;">var</span><span style="background-color: #1f1f1f; color: #cccccc; font-family: Consolas, "Courier New", monospace; font-size: 14px; white-space: pre;"> </span><span style="background-color: #1f1f1f; color: #9cdcfe; font-family: Consolas, "Courier New", monospace; font-size: 14px; white-space: pre;">graphClient</span><span style="background-color: #1f1f1f; color: #cccccc; font-family: Consolas, "Courier New", monospace; font-size: 14px; white-space: pre;"> </span><span style="background-color: #1f1f1f; color: #d4d4d4; font-family: Consolas, "Courier New", monospace; font-size: 14px; white-space: pre;">=</span><span style="background-color: #1f1f1f; color: #cccccc; font-family: Consolas, "Courier New", monospace; font-size: 14px; white-space: pre;"> </span><span style="background-color: #1f1f1f; color: #569cd6; font-family: Consolas, "Courier New", monospace; font-size: 14px; white-space: pre;">new</span><span style="background-color: #1f1f1f; color: #cccccc; font-family: Consolas, "Courier New", monospace; font-size: 14px; white-space: pre;"> </span><span style="background-color: #1f1f1f; color: #4ec9b0; font-family: Consolas, "Courier New", monospace; font-size: 14px; white-space: pre;">GraphServiceClient</span><span style="background-color: #1f1f1f; color: #cccccc; font-family: Consolas, "Courier New", monospace; font-size: 14px; white-space: pre;">(</span><span style="background-color: #1f1f1f; color: #9cdcfe; font-family: Consolas, "Courier New", monospace; font-size: 14px; white-space: pre;">tokenProvider</span><span style="background-color: #1f1f1f; color: #cccccc; font-family: Consolas, "Courier New", monospace; font-size: 14px; white-space: pre;">);</span></p><div style="background-color: #1f1f1f; font-family: Consolas, "Courier New", monospace; font-size: 14px; line-height: 19px; white-space: pre;"><div style="color: #cccccc;"><span style="color: #6a9955;">//</span><span style="color: #6a9955;">Url for below looks like </span><span style="color: #6a9955;">/v1.0/sites/{siteGraphid}}/lists/{listGuid}/items?$select=contentType&$expand=driveItem HTTP/1.1</span></div><div style="color: #cccccc;"><span style="color: #569cd6;">var</span> <span style="color: #9cdcfe;">listItems</span> <span style="color: #d4d4d4;">=</span> <span style="color: #569cd6;">await</span> <span style="color: #9cdcfe;">graphClient</span></div><div style="color: #cccccc;"> .<span style="color: #9cdcfe;">Sites</span>[<span style="color: #9cdcfe;">siteGraphId</span>]</div><div><span style="color: #cccccc;"> .</span><span style="color: #9cdcfe;">Lists</span><span style="color: #cccccc;">[</span><span style="color: #ce9178;">listGuid</span><span style="color: #cccccc;">] </span><span style="color: #6a9955;">//</span><span style="color: #6a9955;"> Need to get the ListId from DriveId</span></div><div style="color: #cccccc;"> .<span style="color: #9cdcfe;">Items</span></div><div style="color: #cccccc;"> .<span style="color: #dcdcaa;">GetAsync</span>(req<span style="color: #d4d4d4;">=></span> { </div><div style="color: #cccccc;"> <span style="color: #9cdcfe;">req</span>.<span style="color: #9cdcfe;">QueryParameters</span>.<span style="color: #9cdcfe;">Expand</span> <span style="color: #d4d4d4;">=</span> <span style="color: #569cd6;">new</span> []{<span style="color: #ce9178;">"</span><span style="color: #ce9178;">driveItem</span><span style="color: #ce9178;">"</span>};</div><div style="color: #cccccc;"> <span style="color: #9cdcfe;">req</span>.<span style="color: #9cdcfe;">QueryParameters</span>.<span style="color: #9cdcfe;">Select</span> <span style="color: #d4d4d4;">=</span> <span style="color: #569cd6;">new</span> []{<span style="color: #ce9178;">"</span><span style="color: #ce9178;">contentType</span><span style="color: #ce9178;">"</span>};</div><div style="color: #cccccc;"> });</div><div style="color: #cccccc;"><span style="color: #9cdcfe;">listItems</span></div><div style="color: #cccccc;">.<span style="color: #9cdcfe;">Value</span></div><div style="color: #cccccc;">.<span style="color: #dcdcaa;">Where</span>(item<span style="color: #d4d4d4;">=></span><span style="color: #9cdcfe;">item</span>.<span style="color: #9cdcfe;">ContentType</span>.<span style="color: #9cdcfe;">Name</span> <span style="color: #d4d4d4;">==</span> <span style="color: #ce9178;">"</span><span style="color: #ce9178;">Document</span><span style="color: #ce9178;">"</span>)</div><div style="color: #cccccc;">.<span style="color: #dcdcaa;">ToList</span>()</div><div style="color: #cccccc;">.<span style="color: #dcdcaa;">ForEach</span>(item<span style="color: #d4d4d4;">=></span><span style="color: #9cdcfe;">Console</span>.<span style="color: #dcdcaa;">WriteLine</span>(<span style="color: #ce9178;">$"</span><span style="color: #ce9178;">FolderName:</span><span style="color: #ce9178;">{</span><span style="color: #9cdcfe;">item</span>.<span style="color: #9cdcfe;">DriveItem</span>.<span style="color: #9cdcfe;">ParentReference</span>.<span style="color: #9cdcfe;">Name</span><span style="color: #ce9178;">}</span><span style="color: #ce9178;">,File Graph Id:</span><span style="color: #ce9178;">{</span><span style="color: #9cdcfe;">item</span>.<span style="color: #9cdcfe;">DriveItem</span>.<span style="color: #9cdcfe;">Id</span><span style="color: #ce9178;">}</span><span style="color: #ce9178;">, File Name: </span><span style="color: #ce9178;">{</span><span style="color: #9cdcfe;">item</span>.<span style="color: #9cdcfe;">DriveItem</span>.<span style="color: #9cdcfe;">Name</span><span style="color: #ce9178;">}</span><span style="color: #ce9178;">"</span>));</div></div><div style="text-align: left;"><br /></div><div style="text-align: left;">The drive Id needs to be translated to ListId to get this working. The QueryParameters.Select is used to limit the data to mainly 2 elements: driveItem and contentType.</div><h2 style="text-align: left;">Full code</h2><div>The above code gives the idea how the required data can be pulled. Below goes full code that can be run inside Polyglot notebook.</div><div><div style="background-color: #1f1f1f; color: #cccccc; font-family: Consolas, "Courier New", monospace; font-size: 14px; line-height: 19px; white-space: pre;"><div><span style="color: #569cd6;">#</span><span style="color: #569cd6;">r</span><span style="color: #569cd6;"> </span><span style="color: #ce9178;">"nuget:Microsoft.Graph"</span></div><div><span style="color: #569cd6;">#</span><span style="color: #569cd6;">r</span><span style="color: #569cd6;"> </span><span style="color: #ce9178;">"nuget:Azure.Identity"</span></div><div><span style="color: #569cd6;">#</span><span style="color: #569cd6;">r</span><span style="color: #569cd6;"> </span><span style="color: #ce9178;">"nuget:CsvHelper"</span></div><div><span style="color: #569cd6;">using</span> <span style="color: #4ec9b0;">Microsoft</span>.<span style="color: #4ec9b0;">DotNet</span>.<span style="color: #4ec9b0;">Interactive</span>;</div><div><span style="color: #569cd6;">using</span> <span style="color: #4ec9b0;">Microsoft</span>.<span style="color: #4ec9b0;">Graph</span>;</div><div><span style="color: #569cd6;">using</span> <span style="color: #4ec9b0;">Microsoft</span>.<span style="color: #4ec9b0;">Graph</span>.<span style="color: #4ec9b0;">Models</span>;</div><div><span style="color: #569cd6;">using</span> <span style="color: #4ec9b0;">Azure</span>.<span style="color: #4ec9b0;">Core</span>;</div><div><span style="color: #569cd6;">using</span> <span style="color: #4ec9b0;">Azure</span>.<span style="color: #4ec9b0;">Identity</span>;</div><div><span style="color: #569cd6;">using</span> <span style="color: #4ec9b0;">System</span>.<span style="color: #4ec9b0;">Collections</span>.<span style="color: #4ec9b0;">Generic</span>;</div><div><span style="color: #569cd6;">using</span> <span style="color: #4ec9b0;">CsvHelper</span>;</div><div><span style="color: #569cd6;">using</span> <span style="color: #4ec9b0;">CsvHelper</span>.<span style="color: #4ec9b0;">Configuration</span>;</div><div><span style="color: #569cd6;">using</span> <span style="color: #4ec9b0;">System</span>.<span style="color: #4ec9b0;">Globalization</span>;</div><div><span style="color: #569cd6;">public</span> <span style="color: #569cd6;">class</span> <span style="color: #4ec9b0;">FooMap</span> : <span style="color: #4ec9b0;">ClassMap</span><(<span style="color: #569cd6;">string</span> FolderName,<span style="color: #569cd6;">string</span> FileGraphId,<span style="color: #569cd6;">string</span> FileName)></div><div>{</div><div> <span style="color: #569cd6;">public</span> <span style="color: #dcdcaa;">FooMap</span>()</div><div> {</div><div> <span style="color: #dcdcaa;">Map</span>(m <span style="color: #d4d4d4;">=></span> <span style="color: #9cdcfe;">m</span>.<span style="color: #9cdcfe;">FolderName</span>).<span style="color: #dcdcaa;">Name</span>(<span style="color: #ce9178;">"</span><span style="color: #ce9178;">FolderName</span><span style="color: #ce9178;">"</span>);</div><div> <span style="color: #dcdcaa;">Map</span>(m <span style="color: #d4d4d4;">=></span> <span style="color: #9cdcfe;">m</span>.<span style="color: #9cdcfe;">FileGraphId</span>).<span style="color: #dcdcaa;">Name</span>(<span style="color: #ce9178;">"</span><span style="color: #ce9178;">FileGraphId</span><span style="color: #ce9178;">"</span>);</div><div> <span style="color: #dcdcaa;">Map</span>(m <span style="color: #d4d4d4;">=></span> <span style="color: #9cdcfe;">m</span>.<span style="color: #9cdcfe;">FileName</span>).<span style="color: #dcdcaa;">Name</span>(<span style="color: #ce9178;">"</span><span style="color: #ce9178;">FileName</span><span style="color: #ce9178;">"</span>);</div><div> }</div><div>}</div><div><span style="color: #4ec9b0;">TokenCredential</span> <span style="color: #9cdcfe;">tokenprovider</span> <span style="color: #d4d4d4;">=</span> <span style="color: #569cd6;">new</span> <span style="color: #4ec9b0;">UsernamePasswordCredential</span>(</div><div> <span style="color: #ce9178;">"{</span><span style="color: #ce9178;">userName@tenant.onmicrosoft.com}</span><span style="color: #ce9178;">"</span>,</div><div> <span style="color: #ce9178;">"{password}</span><span style="color: #ce9178;">"</span>,</div><div> <span style="color: #ce9178;">"</span><span style="color: #ce9178;">{AAD Tenant Id}</span><span style="color: #ce9178;">"</span>,</div><div> <span style="color: #ce9178;">"</span><span style="color: #ce9178;">{AAD App registration id}</span><span style="color: #ce9178;">"</span>);</div><div><span style="color: #569cd6;">var</span> <span style="color: #9cdcfe;">siteGraphId</span><span style="color: #d4d4d4;">=</span><span style="color: #ce9178;">"</span><span style="color: #ce9178;">{3 part graph Id of site}</span><span style="color: #ce9178;">"</span>;</div><div><span style="color: #569cd6;">var</span> <span style="color: #9cdcfe;">listGuid</span> <span style="color: #d4d4d4;">=</span> <span style="color: #ce9178;">"{</span><span style="color: #ce9178;">Guid of List}</span><span style="color: #ce9178;">"</span>;<span style="color: #6a9955;">// Get this using Drive.</span></div><div><span style="color: #9cdcfe;">var</span> <span style="color: #9cdcfe;">graphClient</span> <span style="color: #d4d4d4;">=</span> <span style="color: #569cd6;">new</span> <span style="color: #4ec9b0;">GraphServiceClient</span>(<span style="color: #9cdcfe;">tokenprovider</span>);</div><div><span style="color: #6a9955;">//</span><span style="color: #6a9955;">Url for below looks like /v1.0/sites/{siteGraphid}}/lists/{listGuid}/items?$select=contentType&$expand=driveItem HTTP/1.1</span></div><div><span style="color: #4ec9b0;">ListItemCollectionResponse</span> <span style="color: #9cdcfe;">listItems</span> <span style="color: #d4d4d4;">=</span> <span style="color: #569cd6;">await</span> <span style="color: #9cdcfe;">graphClient</span></div><div> .<span style="color: #9cdcfe;">Sites</span>[<span style="color: #9cdcfe;">siteGraphId</span>]</div><div> .<span style="color: #9cdcfe;">Lists</span>[<span style="color: #9cdcfe;">listGuid</span>]</div><div> .<span style="color: #9cdcfe;">Items</span></div><div> .<span style="color: #dcdcaa;">GetAsync</span>(req<span style="color: #d4d4d4;">=></span> { </div><div> <span style="color: #9cdcfe;">req</span>.<span style="color: #9cdcfe;">QueryParameters</span>.<span style="color: #9cdcfe;">Expand</span> <span style="color: #d4d4d4;">=</span> <span style="color: #569cd6;">new</span> []{<span style="color: #ce9178;">"</span><span style="color: #ce9178;">driveItem</span><span style="color: #ce9178;">"</span>};</div><div> <span style="color: #9cdcfe;">req</span>.<span style="color: #9cdcfe;">QueryParameters</span>.<span style="color: #9cdcfe;">Select</span> <span style="color: #d4d4d4;">=</span> <span style="color: #569cd6;">new</span> []{<span style="color: #ce9178;">"</span><span style="color: #ce9178;">contentType</span><span style="color: #ce9178;">"</span>};</div><div> });</div><div><span style="color: #569cd6;">var</span> <span style="color: #9cdcfe;">finalItems</span> <span style="color: #d4d4d4;">=</span> <span style="color: #569cd6;">new</span> <span style="color: #4ec9b0;">List</span><(<span style="color: #569cd6;">string</span> FolderName,<span style="color: #569cd6;">string</span> FileGraphId,<span style="color: #569cd6;">string</span> FileName)>();</div><div><span style="color: #569cd6;">var</span> <span style="color: #9cdcfe;">pageIterator</span> <span style="color: #d4d4d4;">=</span> <span style="color: #9cdcfe;">PageIterator</span><span style="color: #d4d4d4;"><</span><span style="color: #9cdcfe;">ListItem</span>,<span style="color: #9cdcfe;">ListItemCollectionResponse</span>></div><div>.<span style="color: #9cdcfe;">CreatePageIterator</span>(<span style="color: #9cdcfe;">graphClient</span>,<span style="color: #9cdcfe;">listItems</span>,(<span style="color: #9cdcfe;">item</span>)<span style="color: #d4d4d4;">=</span><span style="color: #d4d4d4;">></span>{</div><div> <span style="color: #c586c0;">if</span>(<span style="color: #9cdcfe;">item</span>.<span style="color: #9cdcfe;">ContentType</span>.<span style="color: #9cdcfe;">Name</span> <span style="color: #d4d4d4;">==</span> <span style="color: #ce9178;">"</span><span style="color: #ce9178;">Document</span><span style="color: #ce9178;">"</span>)</div><div> {</div><div> <span style="color: #9cdcfe;">finalItems</span>.<span style="color: #dcdcaa;">Add</span>((FolderName:<span style="color: #9cdcfe;">item</span>.<span style="color: #9cdcfe;">DriveItem</span>.<span style="color: #9cdcfe;">ParentReference</span>.<span style="color: #9cdcfe;">Name</span>,FileGraphId:<span style="color: #9cdcfe;">item</span>.<span style="color: #9cdcfe;">DriveItem</span>.<span style="color: #9cdcfe;">Id</span>,FileName:<span style="color: #9cdcfe;">item</span>.<span style="color: #9cdcfe;">DriveItem</span>.<span style="color: #9cdcfe;">Name</span>));</div><div> }</div><div> <span style="color: #c586c0;">return</span> <span style="color: #569cd6;">true</span>;</div><div>});</div><div><span style="color: #569cd6;">await</span> <span style="color: #9cdcfe;">pageIterator</span>.<span style="color: #dcdcaa;">IterateAsync</span>();</div><div><span style="color: #569cd6;">using</span>(<span style="color: #569cd6;">var</span> <span style="color: #9cdcfe;">outputFileWriter</span><span style="color: #d4d4d4;">=</span> <span style="color: #569cd6;">new</span> <span style="color: #4ec9b0;">StreamWriter</span>(<span style="color: #ce9178;">@"</span><span style="color: #ce9178;">c:\temp\export.csv</span><span style="color: #ce9178;">"</span>))</div><div>{</div><div> <span style="color: #569cd6;">using</span> (<span style="color: #569cd6;">var</span> <span style="color: #9cdcfe;">csv</span><span style="color: #d4d4d4;">=</span><span style="color: #569cd6;">new</span> <span style="color: #4ec9b0;">CsvWriter</span>(<span style="color: #9cdcfe;">outputFileWriter</span>,<span style="color: #9cdcfe;">CultureInfo</span>.<span style="color: #9cdcfe;">InvariantCulture</span>))</div><div> {</div><div> <span style="color: #9cdcfe;">csv</span>.<span style="color: #9cdcfe;">Context</span>.<span style="color: #dcdcaa;">RegisterClassMap</span><<span style="color: #4ec9b0;">FooMap</span>>();</div><div> <span style="color: #9cdcfe;">csv</span>.<span style="color: #dcdcaa;">WriteRecords</span>(<span style="color: #9cdcfe;">finalItems</span>);</div><div> }</div><div>}</div><div><span style="color: #9cdcfe;">finalItems</span>.<span style="color: #dcdcaa;">ForEach</span>((item)<span style="color: #d4d4d4;">=></span><span style="color: #9cdcfe;">Console</span>.<span style="color: #dcdcaa;">WriteLine</span>(<span style="color: #ce9178;">$"</span><span style="color: #ce9178;">FolderName:</span><span style="color: #ce9178;">{</span><span style="color: #9cdcfe;">item</span>.<span style="color: #9cdcfe;">FolderName</span><span style="color: #ce9178;">}</span><span style="color: #ce9178;">,File Graph Id:</span><span style="color: #ce9178;">{</span><span style="color: #9cdcfe;">item</span>.<span style="color: #9cdcfe;">FileGraphId</span><span style="color: #ce9178;">}</span><span style="color: #ce9178;">, File Name: </span><span style="color: #ce9178;">{</span><span style="color: #9cdcfe;">item</span>.<span style="color: #9cdcfe;">FileName</span><span style="color: #ce9178;">}</span><span style="color: #ce9178;">"</span>))</div><div><span style="color: #6a9955;">Each(item=>Console.WriteLine($"FolderName:{item.DriveItem.ParentReference.Name},File Graph Id:{item.DriveItem.Id}, File Name: {item.DriveItem.Name}"));</span></div></div></div><div style="text-align: left;"><br /></div><div style="text-align: left;">The authentication used here is the ROPC username password. It needs to be changed based on best practices. It uses a nuget package for writing a list as a CSV file.</div><h2 style="text-align: left;">References</h2><div><a href="https://stackoverflow.com/questions/54864198/list-all-files-recursively-in-microsoft-graph">https://stackoverflow.com/questions/54864198/list-all-files-recursively-in-microsoft-graph</a></div>Joymonhttp://www.blogger.com/profile/12014569049635491702noreply@blogger.com0tag:blogger.com,1999:blog-3053230616754516408.post-5803069200756628342023-10-24T10:56:00.001-04:002023-10-24T10:56:00.137-04:00Enable Python preview in Polyglot notebook<p style="text-align: justify;">It is a big difference if anyone ever tried the Jupyter Notebook. Once we experience, we need that experience everywhere including .Net. That experience was there in terms of a Jupyter kernel for .Net<span face="sans-serif" style="background-color: #f8f9fa; color: #202122; font-size: 14px;">¹</span>. That requires Jupyter installed in the machine already. Finally, more native experience came for .Net in the form of a VS Code extension. That was called .Net Interactive Notebooks<span face="sans-serif" style="background-color: #f8f9fa; color: #202122; font-size: 14px;">²</span> which uses .Net Interactive (formerly Try.Net) behind the scene. That started supporting many languages such as PowerShell, JavaScript, SQL etc... which is kinda out of the .Net world. Finally, in Nov 2022, .Net Interactive was renamed to Polyglot notebooks<span face="sans-serif" style="background-color: #f8f9fa; color: #202122; font-size: 14px;">³</span> to reflect its purpose.</p><h2 style="text-align: left;">Is Python supported by Polyglot Notebooks?</h2><p>The Polyglot notebooks didn't have the support for Python. We had to switch to Jypyter to work with Python and come back. Now Polyglot Notebooks has Python support as a preview.</p><h3 style="text-align: left;">How to enable Python in Polyglot Notebook?</h3><div>Obviously, we should have Polyglot notebooks up and running. The first step to enable Python preview is that we need to install Jupyter on the machine and make sure the Python kernel spec is available. Run the below command to make sure it is there.<div class="separator" style="clear: both; text-align: center;"><a href="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/10/24/00-jupyter-kernel.png?raw=true" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="214" data-original-width="1163" height="118" src="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/10/24/00-jupyter-kernel.png?raw=true" width="640" /></a></div>Once this is done, we can see 2 Kernel options at the Notebook level in Visual Studio Code.<div class="separator" style="clear: both; text-align: center;"><a href="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/10/24/01-dual-kernel.png?raw=true" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="207" data-original-width="1454" height="91" src="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/10/24/01-dual-kernel.png?raw=true" width="640" /></a></div><br /></div><div>The enabling preview feature requires the below command to be run on the notebook cell.</div><div><br /></div><div><div style="background-color: #1f1f1f; color: #cccccc; font-family: Consolas, "Courier New", monospace; font-size: 14px; line-height: 19px; white-space: pre;"><span style="color: #6a9955;">#!</span><span style="color: #c586c0;">connect</span><span style="color: #6a9955;"> </span><span style="color: #b5cea8;">jupyter</span><span style="color: #6a9955;"> </span><span style="color: #569cd6;">--kernel-name</span><span style="color: #6a9955;"> </span><span style="color: #b5cea8;">pythonkernel</span><span style="color: #6a9955;"> </span><span style="color: #569cd6;">--kernel-spec</span><span style="color: #6a9955;"> </span><span style="color: #b5cea8;">python3</span></div></div><div><br /></div><div>The command output shows it is in preview mode.</div><div class="separator" style="clear: both; text-align: center;"><a href="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/10/24/02-Python-preview-enable.png?raw=true" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img alt="Command is - #!connect jupyter --kernel-name pythonkernel --kernel-spec python3" border="0" data-original-height="349" data-original-width="1420" height="157" src="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/10/24/02-Python-preview-enable.png?raw=true" width="640" /></a></div><br /><div>Once we add support, we can start having Python cells alongside C#. The cell kernel type needs to be the name we have given in the <i>connect</i> command. Below is the final state. Here we can learn C#<span style="background-color: #f8f9fa; color: #202122; font-family: sans-serif; font-size: 14px;">⁴</span> and Python side by side.</div><div><div class="separator" style="clear: both; text-align: center;"><a href="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/10/24/03-working.png?raw=true" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="544" data-original-width="1428" height="244" src="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/10/24/03-working.png?raw=true" width="640" /></a></div><br /></div><div>We can even set up R language in Polyglot Notebooks<span style="background-color: #f8f9fa; color: #202122; font-family: sans-serif; font-size: 14px;">⁵</span>. Hereafter don't waste time creating an entire project to confirm small language or platform features.</div><h2 style="text-align: left;">References</h2><p><span face="sans-serif" style="background-color: #f8f9fa; color: #202122; font-size: 14px;">¹ - </span><a href="https://devblogs.microsoft.com/dotnet/net-core-with-juypter-notebooks-is-here-preview-1/">https://devblogs.microsoft.com/dotnet/net-core-with-juypter-notebooks-is-here-preview-1/</a></p><p><span face="sans-serif" style="background-color: #f8f9fa; color: #202122; font-size: 14px;">² - </span><a href="https://hanselman.com/blog/announcing-net-interactive-try-net-includes-net-notebooks-and-more">https://hanselman.com/blog/announcing-net-interactive-try-net-includes-net-notebooks-and-more</a></p><p><span face="sans-serif" style="background-color: #f8f9fa; color: #202122; font-size: 14px;">³ - </span><a href="https://devblogs.microsoft.com/dotnet/dotnet-interactive-notebooks-is-now-polyglot-notebooks/">https://devblogs.microsoft.com/dotnet/dotnet-interactive-notebooks-is-now-polyglot-notebooks/</a></p><p><span style="background-color: #f8f9fa; color: #202122; font-family: sans-serif; font-size: 14px;">⁴ - </span><a href="https://techcommunity.microsoft.com/t5/educator-developer-blog/using-visual-studio-notebooks-for-learning-c/ba-p/3580015">https://techcommunity.microsoft.com/t5/educator-developer-blog/using-visual-studio-notebooks-for-learning-c/ba-p/3580015</a></p><p><span style="background-color: #f8f9fa; color: #202122; font-family: sans-serif; font-size: 14px;">⁵ - </span><a href="https://github.com/dotnet/interactive/blob/main/docs/jupyter-in-polyglot-notebooks.md">https://github.com/dotnet/interactive/blob/main/docs/jupyter-in-polyglot-notebooks.md</a></p>Joymonhttp://www.blogger.com/profile/12014569049635491702noreply@blogger.com0tag:blogger.com,1999:blog-3053230616754516408.post-63461468878406838032023-10-18T14:00:00.004-04:002023-10-18T14:00:00.141-04:00[Video] SharePoint Online - Prefer PnP.Core over PnP.Framework<p>This is a transcript of a video published on the blog's <a href="https://www.youtube.com/@joymonscode" rel="nofollow" target="_blank">YouTube channel</a> except for one section. That section was omitted to keep the video under 5 minutes. This may be helpful for fellow developers who cannot view videos during their work.</p><div class="separator" style="clear: both; text-align: center;"><iframe allowfullscreen="" class="BLOG_video_class" height="266" src="https://www.youtube.com/embed/0JzfGrWVxWQ" width="480" youtube-src-id="0JzfGrWVxWQ"></iframe></div><br /><p></p><p>Before discussing about PnP SDKs, please note that the first preference to interact with SharePoint Online should <a href="https://devblogs.microsoft.com/microsoft365dev/net-standard-version-of-sharepoint-online-csom-apis/#:~:text=Always%20use%20Microsoft%20Graph%20APIs,the%20SharePoint%20Online%20CSOM%20APIs." rel="nofollow" target="_blank">always be Graph SDK</a>. That is the official stand from Microsoft. In case Graph API doesn't have the feature, go for SharePoint APIs or CSOM APIs. There are 2 different SDKs or nuget packages to interact with CSOM API from .Net applications. PnP.Framework and PnP.Core to name the nugets. Today we are talking about these 2 PnP SDKs what SDK we should choose and why one over the other?</p><p>As per the documentation the <a href="https://github.com/pnp/pnpcore#whats-the-relationship-with-the-existing-pnp-framework-library" rel="nofollow" target="_blank">PnP.Core is the successor of PnP.Framework</a>. In that sense PnP.Core should be our clear preference. One benefit they are advertising is that it can seamlessly switch between the modern Graph API and old SharePoint CSOM APIs. That will hide the API complexities from the developer and the developer needs to learn only PnP.Core. But there are more reasons than that to choose PnP.Core.</p><h2 style="text-align: left;">Number of lines</h2><p>First of all PnP.Core requires less number of lines to achieve a task. Take the example of Getting the title of the web.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/10/10-pnpcore-over-pnpframework/06-number-of-lines.png?raw=true" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="812" data-original-width="1665" height="312" src="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/10/10-pnpcore-over-pnpframework/06-number-of-lines.png?raw=true" width="640" /></a></div><p></p><p>The left side is PnP.Framework code and the right side show PnP.Core. We can clearly see that the PnP.Core works with one less line. Compare this in an enterprise application with tens of thousands of lines.</p><h2 style="text-align: left;">Ease of use</h2><div>Now let us look at the ease of use in achieving a use case. The example we are taking is the renaming of a folder in the document library. PnP.Core has a direct <a href="https://pnp.github.io/pnpcore/using-the-sdk/folders-intro.html#renaming-folders" rel="nofollow" target="_blank">Rename method</a> to do that. But PnP.Framework uses the move method with the new name.</div><h2 style="text-align: left;">Support for Dependency injection</h2><p>The PnP.Core SDK has out-of-the-box dependency injection support. If we are using PnP.Framework, we need to write extra code to get the .net core model dependency injection.</p><h2 style="text-align: left;">Size of the request payload</h2><p>The PnP.Framework uses one particular URL almost always. It is _vti_bin/client.svc/ProcessQuery. This is the Fiddler screenshot of PnP.Framework making HTTP call to get a folder by relative Url. It uses HTTP POST to get data instead of HTTP GET. The request payload is XML format.Luckily response is in JSON. The XML requires more bytes than JSON for sending the same information. The different operations are achieved by changing the method attribute in the request payload. </p><div class="separator" style="clear: both; text-align: center;"><a href="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/10/10-pnpcore-over-pnpframework/09-02-payload%20pnpframework-request.png?raw=true" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="915" data-original-width="1910" height="307" src="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/10/10-pnpcore-over-pnpframework/09-02-payload%20pnpframework-request.png?raw=true" width="640" /></a></div><br /><p></p><p>Now let us see how the same use case is done by PnP.Core. As we can see it uses HTTP GET with user friendly URL. No request payload at all. The response payload is relatively smaller than the PnP.Framework</p><p>Here is the side-by-side comparison between the two when querying for a folder by its relative Url. Left is PnP.Framework and right side is the PnP.Core.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/10/10-pnpcore-over-pnpframework/09-08-payload-sidebyside.png?raw=true" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="918" data-original-width="1922" height="306" src="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/10/10-pnpcore-over-pnpframework/09-08-payload-sidebyside.png?raw=true" width="640" /></a></div><br /><p></p><h2 style="text-align: left;">Logging</h2><div>Now coming to the logging aspects, PnP.Core is already following the .Net programming model and honoring the ILogger API. By adding the required configuration in the appsettings.json file we can get the logs into the required destination.</div><h2 style="text-align: left;">Observability</h2><div>Observability is important to run a production workload. Tools such as Azure Application Insights normally log the URLs and their HTTP verbs. That will help us to understand the system behavior and also to troubleshoot issues. We have already seen that the Url is used by the PnP.Framework is mostly the same. Adding to that, it always uses HTTP POST with parameters to distinguish the operation which makes. We cannot distinguish the operation type by the HTTP verbs.<div style="text-align: center;"><a href="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/10/10-pnpcore-over-pnpframework/09-01-payload-pnpframework-request-headers.png?raw=true"><img border="0" data-original-height="976" data-original-width="1917" height="326" src="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/10/10-pnpcore-over-pnpframework/09-01-payload-pnpframework-request-headers.png?raw=true" width="640" /></a></div><br /></div><h2 style="text-align: left;">Error handling</h2><div>In a normal web system, when an error happens it will return respective HTTP status codes. Never Http 200. Look a the fiddler trace when querying with non existing folder path from PnP.Framework. it always returns HTTP 200. In case it errors out, the error message will be inside. Production observability systems can never detect errors like this and proactively alert the dev team. <div class="separator" style="clear: both; text-align: center;"><a href="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/10/10-pnpcore-over-pnpframework/11-01-observ-pnpfwk-status.png?raw=true" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="924" data-original-width="1898" height="312" src="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/10/10-pnpcore-over-pnpframework/11-01-observ-pnpfwk-status.png?raw=true" width="640" /></a></div><br /></div><div>Otherside, PnP.Core error out with proper HTTP status code. In case the folder URL is not present, we get HTTP 404 back, and will be handled properly by the production monitoring systems.<div class="separator" style="clear: both; text-align: center;"><a href="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/10/10-pnpcore-over-pnpframework/11-05-observ-pnpcore.png?raw=true" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="923" data-original-width="1901" height="311" src="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/10/10-pnpcore-over-pnpframework/11-05-observ-pnpcore.png?raw=true" width="640" /></a></div></div><h2 style="text-align: left;">Contribution to Library</h2><div>The PnP libraries are open-source and maintained by the community. Microsoft is not giving any warranty or support for these libraries. In case we end up facing an issue and need to get fixed quickly we need to fix ourselves and give a pull request.<div class="separator" style="clear: both; text-align: center;"><a href="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/10/10-pnpcore-over-pnpframework/15-01-no-msft-support.png?raw=true" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="704" data-original-width="1264" height="356" src="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/10/10-pnpcore-over-pnpframework/15-01-no-msft-support.png?raw=true" width="640" /></a></div><div><br /></div>Look at the source code of PnP.Core and PnP.Framework. The PnP.Core is relatively simple to understand. Meaning easy for us to fix the bugs in that library. It simply prepares the Url uses an internal method to execute that Url and returns the result.<div class="separator" style="clear: both; text-align: center;"><a href="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/10/10-pnpcore-over-pnpframework/15-02-simple-source.png?raw=true" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="723" data-original-width="1498" height="309" src="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/10/10-pnpcore-over-pnpframework/15-02-simple-source.png?raw=true" width="640" /></a></div></div><h2 style="text-align: left;">Can we completely avoid PnP.Framework?</h2><div>Not really. The PnP.Core is not a feature-complete SDK. There are features that are not available in PnP.Core.. In such situations, we need to use PnP.Framework. One place they are telling PnP.Core will eventually replace PnP.Framework. However, when a feature request was created to bridge the gap in CreateCopyJobs-related tasks, they informed PnP.Core is never going to implement that feature. In such situations we need PnP.Framework forever.<div class="separator" style="clear: both; text-align: center;"><a href="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/10/10-pnpcore-over-pnpframework/18-createcopyjobs-rejected.png?raw=true" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="865" data-original-width="1883" height="294" src="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/10/10-pnpcore-over-pnpframework/18-createcopyjobs-rejected.png?raw=true" width="640" /></a></div>Thanks for reading. Comments are welcome.</div>Joymonhttp://www.blogger.com/profile/12014569049635491702noreply@blogger.com0tag:blogger.com,1999:blog-3053230616754516408.post-79588425547978967822023-10-03T10:00:00.005-04:002023-10-03T11:29:13.781-04:00[Video] Configuring CodeGPT to use Azure OpenAI key<p>If there is a way to get GitHub Copilot, use it and don't look back. But if we don't have the luxury to afford the license or there are other reasons but still want to leverage AI continue reading this post.</p><h2 style="text-align: left;">CodeGPT VS Code extension</h2><p>This is not a complete replacement for Copilot but it can give some level of intelligence and speed up development. Checking code quality, and explaining the code are some to use.</p><p>They have their own licensing model where the AI comes from them or we can use their extension to use our preferred AI providers. If we don't have GitHub Copilot but have an Azure subscription with Azure Open AI service we can configure the CodeGPT extension using the key from Azure Open AI.</p><h2 style="text-align: left;">Why this post and a video just to configure?</h2><div>It may sound easy. Why a post to do the configuration? The reason is that the provided documentation¹ is not clear and the extension configuration screen leads us to the wrong URL to configure.</div><div>eg: the extension configuration screen is not showing the <i>/openai/</i> in the URL also the fragment is to be added after the model name. But in order to configure we need all.</div><div><br /></div><div>The endpoint of the Azure OpenAI service will look like below.</div><div><br /></div><div><i>https://<your name>.openai.azure.com</i></div><div><i><br /></i></div><div>But this is not the URL to be used to configure. The URL to configure is as follows</div><div><br /></div><div><i>https://<your name>.openai.azure.com/openai/deployments/<model name>/chat/completions?api-version=2023-07-01-preview</i></div><div><br /></div><div>Yeah, there is a video also made to make it simple.</div><div class="separator" style="clear: both; text-align: center;"><iframe allowfullscreen="" class="BLOG_video_class" height="266" src="https://www.youtube.com/embed/JTeCbkpuCvY" width="480" youtube-src-id="JTeCbkpuCvY"></iframe></div><br /><div><br /></div><h2 style="text-align: left;">References</h2><div>¹ - <a href="https://docs.codegpt.co/docs/tutorial-basics/installation#microsoft-azure">https://docs.codegpt.co/docs/tutorial-basics/installation#microsoft-azure</a></div>Joymonhttp://www.blogger.com/profile/12014569049635491702noreply@blogger.com0tag:blogger.com,1999:blog-3053230616754516408.post-16994540992022274902023-09-26T21:30:00.003-04:002023-09-26T21:30:00.146-04:00SharePoint Online - PnP.Core sample to check CopyJobProgress via Azure Storage Queue SDK for .Net<p>This is the continuation of the <a href="https://joymonscode.blogspot.com/2023/09/sharepoint-online-check-copyjobprogress.html" rel="nofollow" target="_blank">last post</a> in this blog related to checking the status of CopyJobs in SharePoint Online. The last post has code snippets using PnP.Framework. It is not a complete compilable and working sample. This post is to introduce a functional example. The only difference is that this sample is using PnP.Core SDK.</p><p>If anyone wondering what is PnP.Framework and PnP.Core, please have a look at the <a href="https://joymonscode.blogspot.com/2022/05/dilemma-of-choosing-net-sdk-to-interact.html" rel="nofollow" target="_blank">dilemma of choosing .Net SDK to interact with SharePoint</a> Online.</p><blockquote><p>Sample located at <a href="https://github.com/dotnet-demos/sharepoint-createcopyjob-azure-queue-tracking">https://github.com/dotnet-demos/sharepoint-createcopyjob-azure-queue-tracking</a>.</p></blockquote><h2 style="text-align: left;">Points of interest</h2><div>Though the code has a lot of comments, below are the areas that may bring some questions.</div><h3 style="text-align: left;">Getting the job status</h3><div>There is no way to get the Copy Job status in PnP.Core SDK. Even there is no plan to implement that feature. Feature request <a href="https://github.com/pnp/pnpcore/issues/1277" rel="nofollow" target="_blank">#1277</a> was closed.</div><div><br /></div><div>So either we need to switch to PnP.Framework SDK or make a direct HTTP call. In the sample, we can see it is making a direct HTTP call.</div><h3 style="text-align: left;">Determining the JSON request body for direct HTTP calls</h3><div>If we refer to any documentation related to SharePoint migration API, it is referring to the SDK which is nothing but .Net classes and methods. No HTTP request-response level documentation is available. The PnP.Framework SDK uses XML to craft the request payload and the response is JSON. That request XML is the hardest thing in the world to understand. Luckily the PnP.Core SDK uses JSON for request and response. Though there is <a href="https://learn.microsoft.com/en-us/sharepoint/dev/sp-add-ins/complete-basic-operations-using-sharepoint-rest-endpoints" rel="nofollow" target="_blank">a tutorial on basic operations to SharePoint REST</a> endpoints, it is difficult to craft the JSON request to do a simple operation such as GetMigrationJobStatus(guide jobId).</div><h4 style="text-align: left;">Raw HTTP call to GetMigrationJobStatus(guide jobId).</h4><div>The official documentation is there in 2 places. But in <a href="https://learn.microsoft.com/nl-nl/dotnet/api/microsoft.sharepoint.client.site.getmigrationjobstatus?view=sharepoint-csom" rel="nofollow" target="_blank">both</a> <a href="https://learn.microsoft.com/en-us/previous-versions/office/sharepoint-csom/mt143033(v=office.15)" rel="nofollow" target="_blank">places</a>, it is talking about .Net SDK usage, not the HTTP-level request response.</div><div><br /></div><div>First, we have to decide what should be the HTTP method. Common sense tells us this should be GET as its read operation. Url should be something like <i>_api/site/GetMigrationJobStatus/{JobId}</i>. </div><div>We are totally wrong. It should be HTTP POST.</div><div><br /></div><div>Now how should we build the POST body to include the jobId? Just the id in the POST body or we should have an object as below?</div><div><i>{</i></div><div><i>"jobId":"{jobId}"</i></div><div><i>}</i></div><div><br /></div><div>Nothing is gonna work except the below one.</div><div><br /></div><div><div><i>{</i></div><div><i>"id":"{jobId}"</i></div><div><i>}</i></div></div><div><br /></div><div>Refer to the <i>GetJobStatus()</i> in the example.</div><div><br /></div><div>Have fun with SharePoint. Also please give PR or create an issue in case of any improvements to be made.</div>Joymonhttp://www.blogger.com/profile/12014569049635491702noreply@blogger.com0tag:blogger.com,1999:blog-3053230616754516408.post-43927625686807836092023-09-19T22:00:00.004-04:002023-09-26T19:12:23.972-04:00SharePoint Online - Check CopyJobProgress via Azure Storage Queue SDK for .Net<p>This post assumes some knowledge about SharePoint Online, it's file storage model via DocumentLibrary ie Drive ie List and <i>CreateCopyJobs</i> API used to bulk move drive items. Also Azure storage queue and how to interact with it via .Net SDK</p><h2 style="text-align: left;">Scenario</h2><div>We must copy all the folders of one SharePoint drive sitting in one SharePoint Online site to drive sitting in another site. The destination site already has the drive created. It needs to be done efficiently i.e. without downloading the files from one SharePoint drive and upload to another.</div><div><br /></div><div><blockquote>There is no drive copy mechanism available in either Graph API as well as legacy SharePoint API (CSOM). Please comment if one exists. </blockquote></div><div>One of the promising ways is to use the <i><a href="https://learn.microsoft.com/en-us/sharepoint/dev/apis/spod-copy-move-api" rel="nofollow" target="_blank">CreateCopyJobs API</a></i> present in SharePoint API via PnP.Framework SDK for .Net. This API accepts a list of drive item URLs, and a destination URL then returns one or more Jobs. This job is technically de-queued by Microsoft worker machines and does the actual copy job. They may have backdoors to copy the files efficiently. The returned value is of type <i><a href="https://learn.microsoft.com/en-us/dotnet/api/microsoft.sharepoint.client.copymigrationinfo?view=sharepoint-csom" rel="nofollow" target="_blank">CopyMigrationInfo</a></i>. This has a JobId of type Guid. An <a href="https://learn.microsoft.com/en-us/dotnet/api/microsoft.sharepoint.client.copymigrationinfo.jobqueueuri?view=sharepoint-csom#microsoft-sharepoint-client-copymigrationinfo-jobqueueuri" rel="nofollow" target="_blank">Url to random Azure Storage Queue with SAS token</a>, <a href="https://learn.microsoft.com/en-us/dotnet/api/microsoft.sharepoint.client.copymigrationinfo.encryptionkey?view=sharepoint-csom#microsoft-sharepoint-client-copymigrationinfo-encryptionkey" rel="nofollow" target="_blank">a key to decrypt </a>messages, and the IDs of drive items it is working on.</div><div><blockquote>Note that the Azure Storage Queue is not to store the work messages. It has the log messages that are generated as part of processing. </blockquote></div><div>Each Job can be polled via <i><a href="https://learn.microsoft.com/en-us/dotnet/api/microsoft.sharepoint.client.site.getmigrationjobstatus?view=sharepoint-csom#microsoft-sharepoint-client-site-getmigrationjobstatus(system-guid)" rel="nofollow" target="_blank">site.GetMigrationJobStatus(jobId)</a></i> to check the high-level status of <a href="https://learn.microsoft.com/en-us/dotnet/api/microsoft.sharepoint.client.migrationjobstate?view=sharepoint-csom" rel="nofollow" target="_blank">Queued, Processing, and None</a>. completed. As per direct Microsoft meetings, <i>None</i> means Completed. Nowhere it is documented.</div><div>None ie Completed means it is a success. Failed also will be considered completed. In order to get a detailed status of whether it failed or not we need to use another API <i><a href="https://learn.microsoft.com/en-us/dotnet/api/microsoft.sharepoint.client.site.getcopyjobprogress?view=sharepoint-csom#microsoft-sharepoint-client-site-getcopyjobprogress(microsoft-sharepoint-client-copymigrationinfo)" rel="nofollow" target="_blank">site.GetCopyJobProgress(jobInfo)</a></i>. This returns <i><a href="https://learn.microsoft.com/en-us/dotnet/api/microsoft.sharepoint.client.copyjobprogress?view=sharepoint-csom" rel="nofollow" target="_blank">CopyJobProgress</a></i> object. Using this <i>GetCopyJobProgress</i> API is complex. It emits Logs property sometimes. </div><div><br /></div><div>So we can look at the CopyMigrationInfo object that we got after the Job creation and its URL to dequeue log messages from Azure Storage Queue.</div><div><br /></div><div>This post is about how to check the status of CopyJobs via the Azure Storage Queue. The second warning here. Without the familiarity of these APIs, it would be difficult to understand the rest.</div><h2 style="text-align: left;">Problem</h2><p>99.99% of how this thing is working is not documented. No working sample from Microsoft to use this <i>CreateCopyJobs </i>API. The PnP.Framework SDK doesn't support dequeue messages from the Azure Storage Queue. Even there is no class to deserialize the output log entry into.</p><h2 style="text-align: left;">Solution</h2><p>Let us start with code as the high-level steps are to de-queue messages and decrypt using the key provided. The encryption algorithm is AES256 with CBC. The IV is base64 encoded and available with the message.</p><div style="background-color: #1f1f1f; color: #cccccc; font-family: Consolas, "Courier New", monospace; font-size: 14px; line-height: 19px; white-space: pre;"><div> <span style="color: #569cd6;">private</span> <span style="color: #4ec9b0;">IEnumerable</span><<span style="color: #4ec9b0;">CopyJobLog</span>> <span style="color: #dcdcaa;">GetCopyJobLogsFromAzureStorageQueue</span>(<span style="color: #4ec9b0;">CopyMigrationInfo</span> <span style="color: #9cdcfe;">info</span>)</div><div> {</div><div> <span style="color: #4ec9b0;">Response</span><<span style="color: #4ec9b0;">QueueMessage</span>[]> <span style="color: #9cdcfe;">messages</span>;</div><div> <span style="color: #4ec9b0;">List</span><<span style="color: #4ec9b0;">CopyJobLog</span>> <span style="color: #9cdcfe;">result</span> <span style="color: #d4d4d4;">=</span> <span style="color: #569cd6;">new</span>();</div><div> <span style="color: #c586c0;">do</span></div><div> {</div><div> <span style="color: #4ec9b0;">QueueClient</span> <span style="color: #9cdcfe;">client</span> <span style="color: #d4d4d4;">=</span> <span style="color: #569cd6;">new</span>(<span style="color: #569cd6;">new</span> <span style="color: #4ec9b0;">Uri</span>(<span style="color: #9cdcfe;">info</span>.<span style="color: #9cdcfe;">JobQueueUri</span>));</div><div> <span style="color: #9cdcfe;">messages</span> <span style="color: #d4d4d4;">=</span> <span style="color: #9cdcfe;">client</span>.<span style="color: #dcdcaa;">ReceiveMessages</span>(<span style="color: #9cdcfe;">maxMessages</span>: <span style="color: #b5cea8;">25</span>);</div><div> <span style="color: #4ec9b0;">IEnumerable</span><<span style="color: #4ec9b0;">CopyJobLog</span>> <span style="color: #9cdcfe;">logs</span> <span style="color: #d4d4d4;">=</span> <span style="color: #dcdcaa;">DecryptAndDeserializeMessages</span>(<span style="color: #9cdcfe;">messages</span>, <span style="color: #9cdcfe;">info</span>.<span style="color: #9cdcfe;">EncryptionKey</span>);</div><div> <span style="color: #9cdcfe;">result</span>.<span style="color: #dcdcaa;">AddRange</span>(<span style="color: #9cdcfe;">logs</span>);</div><div> } <span style="color: #c586c0;">while</span> (<span style="color: #9cdcfe;">messages</span>.<span style="color: #9cdcfe;">Value</span>.<span style="color: #9cdcfe;">Length</span> <span style="color: #d4d4d4;">!=</span> <span style="color: #b5cea8;">0</span>);</div><div> <span style="color: #c586c0;">return</span> <span style="color: #9cdcfe;">result</span>;</div><div> }</div></div><p>Now the code for DecryptAndDeserializeMessages()</p><div style="background-color: #1f1f1f; color: #cccccc; font-family: Consolas, "Courier New", monospace; font-size: 14px; line-height: 19px; white-space: pre;"><div> <span style="color: #569cd6;">private</span> <span style="color: #9cdcfe;">IEnumerable</span><<span style="color: #9cdcfe;">CopyJobLog</span>> <span style="color: #dcdcaa;">DecryptAndDeserializeMessages</span>(<span style="color: #9cdcfe;">Response</span><<span style="color: #9cdcfe;">QueueMessage</span>[]> <span style="color: #9cdcfe;">messages</span>, <span style="color: #569cd6;">byte</span>[] <span style="color: #9cdcfe;">encryptionKey</span>)</div><div> {</div><div> <span style="color: #c586c0;">foreach</span> (<span style="color: #9cdcfe;">Azure</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">Storage</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">Queues</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">Models</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">QueueMessage</span> <span style="color: #9cdcfe;">msg</span> <span style="color: #c586c0;">in</span> <span style="color: #9cdcfe;">messages</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">Value</span>)</div><div> {</div><div> <span style="color: #569cd6;">var</span> <span style="color: #9cdcfe;">base64DecodedArray</span> <span style="color: #d4d4d4;">=</span> <span style="color: #9cdcfe;">Convert</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">FromBase64String</span>(<span style="color: #9cdcfe;">msg</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">Body</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">ToString</span>());</div><div> <span style="color: #569cd6;">var</span> <span style="color: #9cdcfe;">jsonBody</span> <span style="color: #d4d4d4;">=</span> <span style="color: #9cdcfe;">Encoding</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">UTF8</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">GetString</span>(<span style="color: #9cdcfe;">base64DecodedArray</span>);</div><div> <span style="color: #4ec9b0;">AzureJobProgress</span> <span style="color: #9cdcfe;">progress</span> <span style="color: #d4d4d4;">=</span> <span style="color: #9cdcfe;">JsonConvert</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">DeserializeObject</span><<span style="color: #4ec9b0;">AzureJobProgress</span>>(<span style="color: #9cdcfe;">jsonBody</span>);</div><div> <span style="color: #569cd6;">string</span> <span style="color: #9cdcfe;">progressString</span> <span style="color: #d4d4d4;">=</span> <span style="color: #9cdcfe;">progress</span><span style="color: #d4d4d4;">.</span><span style="color: #dcdcaa;">Decrypt</span>(<span style="color: #9cdcfe;">encryptionKey</span>);</div><div> <span style="color: #c586c0;">yield</span> <span style="color: #c586c0;">return</span> <span style="color: #9cdcfe;">JsonConvert</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">DeserializeObject</span><<span style="color: #9cdcfe;">CopyJobLog</span>>(<span style="color: #9cdcfe;">progressString</span>);</div><div> }</div><div> }</div></div><p>Now we need the code for 2 entity classes and a couple of enums. They are below</p><div style="background-color: #1f1f1f; color: #cccccc; font-family: Consolas, "Courier New", monospace; font-size: 14px; line-height: 19px; white-space: pre;"><div> <span style="color: #569cd6;">class</span> <span style="color: #4ec9b0;">AzureJobProgress</span></div><div> {</div><div> <span style="color: #569cd6;">public</span> <span style="color: #569cd6;">string</span> <span style="color: #9cdcfe;">Label</span> { <span style="color: #569cd6;">get</span>; <span style="color: #569cd6;">set</span>; }</div><div> <span style="color: #569cd6;">public</span> <span style="color: #569cd6;">string</span> <span style="color: #9cdcfe;">JobId</span> { <span style="color: #569cd6;">get</span>; <span style="color: #569cd6;">set</span>; }</div><div> <span style="color: #569cd6;">public</span> <span style="color: #569cd6;">string</span> <span style="color: #9cdcfe;">IV</span> { <span style="color: #569cd6;">get</span>; <span style="color: #569cd6;">set</span>; }</div><div> <span style="color: #569cd6;">public</span> <span style="color: #569cd6;">string</span> <span style="color: #9cdcfe;">Content</span> { <span style="color: #569cd6;">get</span>; <span style="color: #569cd6;">set</span>; }</div><br /><div> <span style="color: #569cd6;">public</span> <span style="color: #569cd6;">string</span> <span style="color: #dcdcaa;">Decrypt</span>(<span style="color: #569cd6;">byte</span>[] <span style="color: #9cdcfe;">key</span>)</div><div> {</div><div> <span style="color: #569cd6;">using</span> (<span style="color: #4ec9b0;">Aes</span> <span style="color: #9cdcfe;">aes</span> <span style="color: #d4d4d4;">=</span> <span style="color: #4ec9b0;">Aes</span><span style="color: #d4d4d4;">.</span><span style="color: #dcdcaa;">Create</span>())</div><div> {</div><div> <span style="color: #9cdcfe;">aes</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">Key</span> <span style="color: #d4d4d4;">=</span> <span style="color: #9cdcfe;">key</span>;</div><div> <span style="color: #9cdcfe;">aes</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">IV</span> <span style="color: #d4d4d4;">=</span> <span style="color: #4ec9b0;">Convert</span><span style="color: #d4d4d4;">.</span><span style="color: #dcdcaa;">FromBase64String</span>(<span style="color: #9cdcfe;">IV</span>);</div><div> <span style="color: #9cdcfe;">aes</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">Mode</span> <span style="color: #d4d4d4;">=</span> <span style="color: #4ec9b0;">CipherMode</span><span style="color: #d4d4d4;">.</span><span style="color: #4fc1ff;">CBC</span>;</div><div> <span style="color: #569cd6;">using</span> (<span style="color: #4ec9b0;">ICryptoTransform</span> <span style="color: #9cdcfe;">decipher</span> <span style="color: #d4d4d4;">=</span> <span style="color: #9cdcfe;">aes</span><span style="color: #d4d4d4;">.</span><span style="color: #dcdcaa;">CreateDecryptor</span>(<span style="color: #9cdcfe;">aes</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">Key</span>, <span style="color: #9cdcfe;">aes</span><span style="color: #d4d4d4;">.</span><span style="color: #9cdcfe;">IV</span>))</div><div> {</div><div> <span style="color: #569cd6;">using</span> (<span style="color: #4ec9b0;">MemoryStream</span> <span style="color: #9cdcfe;">ms</span> <span style="color: #d4d4d4;">=</span> <span style="color: #569cd6;">new</span> <span style="color: #4ec9b0;">MemoryStream</span>(<span style="color: #4ec9b0;">Convert</span><span style="color: #d4d4d4;">.</span><span style="color: #dcdcaa;">FromBase64String</span>(<span style="color: #9cdcfe;">Content</span>)))</div><div> {</div><div> <span style="color: #569cd6;">using</span> (<span style="color: #4ec9b0;">CryptoStream</span> <span style="color: #9cdcfe;">cs</span> <span style="color: #d4d4d4;">=</span> <span style="color: #569cd6;">new</span> <span style="color: #4ec9b0;">CryptoStream</span>(<span style="color: #9cdcfe;">ms</span>, <span style="color: #9cdcfe;">decipher</span>, <span style="color: #4ec9b0;">CryptoStreamMode</span><span style="color: #d4d4d4;">.</span><span style="color: #4fc1ff;">Read</span>))</div><div> {</div><div> <span style="color: #569cd6;">using</span> (<span style="color: #4ec9b0;">StreamReader</span> <span style="color: #9cdcfe;">sr</span> <span style="color: #d4d4d4;">=</span> <span style="color: #569cd6;">new</span> <span style="color: #4ec9b0;">StreamReader</span>(<span style="color: #9cdcfe;">cs</span>))</div><div> {</div><div> <span style="color: #c586c0;">return</span> <span style="color: #9cdcfe;">sr</span><span style="color: #d4d4d4;">.</span><span style="color: #dcdcaa;">ReadToEnd</span>();</div><div> }</div><div> }</div><div> }</div><div> }</div><div> }</div><div> }</div><div> }</div></div><p>The last one has more properties and enums.</p><div style="background-color: #1f1f1f; color: #cccccc; font-family: Consolas, "Courier New", monospace; font-size: 14px; line-height: 19px; white-space: pre;"><div><span style="color: #569cd6;">enum</span> <span style="color: #4ec9b0;">CopyJobLogMigrationDirection</span></div><div> {</div><div> <span style="color: #9cdcfe;">Export</span>,</div><div> <span style="color: #9cdcfe;">Import</span></div><div> }</div><div> <span style="color: #569cd6;">enum</span> <span style="color: #4ec9b0;">CopyJobLogEvent</span></div><div> {</div><div> <span style="color: #9cdcfe;">JobQueued</span>,</div><div> <span style="color: #9cdcfe;">JobStart</span>,</div><div> <span style="color: #9cdcfe;">JobLogFileCreate</span>,</div><div> <span style="color: #9cdcfe;">FinishManifestFileUpload</span>,</div><div> <span style="color: #9cdcfe;">JobProgress</span>,</div><div> <span style="color: #9cdcfe;">JobEnd</span>,</div><div> <span style="color: #9cdcfe;">JobFinishedObjectInfo</span>,</div><div> <span style="color: #9cdcfe;">JobWarning</span>,</div><div> }</div><div> <span style="color: #569cd6;">enum</span> <span style="color: #4ec9b0;">CopyJobLogMigrationType</span></div><div> {</div><div> <span style="color: #9cdcfe;">Copy</span>,</div><div> }</div><div> <span style="color: #569cd6;">class</span> <span style="color: #4ec9b0;">CopyJobLog</span></div><div> {</div><div> <span style="color: #569cd6;">public</span> <span style="color: #4ec9b0;">CopyJobLogEvent</span> <span style="color: #9cdcfe;">Event</span> { <span style="color: #569cd6;">get</span>; <span style="color: #569cd6;">set</span>; }</div><div> <span style="color: #569cd6;">public</span> <span style="color: #569cd6;">int</span> <span style="color: #9cdcfe;">TotalRetryCount</span> { <span style="color: #569cd6;">get</span>; <span style="color: #569cd6;">set</span>; }<span style="color: #6a9955;">//Set for Events:JobProgress,</span></div><div> <span style="color: #569cd6;">public</span> <span style="color: #4ec9b0;">DateTime</span> <span style="color: #9cdcfe;">Time</span> { <span style="color: #569cd6;">get</span>; <span style="color: #569cd6;">set</span>; }</div><div> <span style="color: #569cd6;">public</span> <span style="color: #4ec9b0;">Guid</span> <span style="color: #9cdcfe;">JobId</span> { <span style="color: #569cd6;">get</span>; <span style="color: #569cd6;">set</span>; }</div><div> <span style="color: #569cd6;">public</span> <span style="color: #4ec9b0;">Guid</span> <span style="color: #9cdcfe;">SiteId</span> { <span style="color: #569cd6;">get</span>; <span style="color: #569cd6;">set</span>; }</div><div> <span style="color: #569cd6;">public</span> <span style="color: #4ec9b0;">Guid</span> <span style="color: #9cdcfe;">DbId</span> { <span style="color: #569cd6;">get</span>; <span style="color: #569cd6;">set</span>; }</div><div> <span style="color: #569cd6;">public</span> <span style="color: #4ec9b0;">CopyJobLogMigrationType</span> <span style="color: #9cdcfe;">MigrationType</span> { <span style="color: #569cd6;">get</span>; <span style="color: #569cd6;">set</span>; }</div><div> <span style="color: #569cd6;">public</span> <span style="color: #4ec9b0;">CopyJobLogMigrationDirection</span> <span style="color: #9cdcfe;">MigrationDirection</span> { <span style="color: #569cd6;">get</span>; <span style="color: #569cd6;">set</span>; }</div><div> <span style="color: #569cd6;">public</span> <span style="color: #569cd6;">string</span> <span style="color: #9cdcfe;">Url</span> { <span style="color: #569cd6;">get</span>; <span style="color: #569cd6;">set</span>; }</div><div> <span style="color: #569cd6;">public</span> <span style="color: #569cd6;">int</span> <span style="color: #9cdcfe;">FilesCreated</span> { <span style="color: #569cd6;">get</span>; <span style="color: #569cd6;">set</span>; }<span style="color: #6a9955;">//Set for Events:JobProgress,</span></div><div> <span style="color: #569cd6;">public</span> <span style="color: #569cd6;">int</span> <span style="color: #9cdcfe;">BytesProcessed</span> { <span style="color: #569cd6;">get</span>; <span style="color: #569cd6;">set</span>; }<span style="color: #6a9955;">//Set for Events:JobProgress,</span></div><div> <span style="color: #569cd6;">public</span> <span style="color: #569cd6;">int</span> <span style="color: #9cdcfe;">TotalExpectedBytes</span> { <span style="color: #569cd6;">get</span>; <span style="color: #569cd6;">set</span>; }</div><div> <span style="color: #569cd6;">public</span> <span style="color: #569cd6;">int</span> <span style="color: #9cdcfe;">ObjectsProcessed</span> { <span style="color: #569cd6;">get</span>; <span style="color: #569cd6;">set</span>; }<span style="color: #6a9955;">//Set for Events:JobProgress,</span></div><div> <span style="color: #569cd6;">public</span> <span style="color: #569cd6;">int</span> <span style="color: #9cdcfe;">TotalExpectedSPObjects</span> { <span style="color: #569cd6;">get</span>; <span style="color: #569cd6;">set</span>; }<span style="color: #6a9955;">//Set for Events:JobProgress,</span></div><div> <span style="color: #569cd6;">public</span> <span style="color: #569cd6;">int</span> <span style="color: #9cdcfe;">TotalErrors</span> { <span style="color: #569cd6;">get</span>; <span style="color: #569cd6;">set</span>; }<span style="color: #6a9955;">//Set for Events:JobProgress,</span></div><div> <span style="color: #569cd6;">public</span> <span style="color: #569cd6;">int</span> <span style="color: #9cdcfe;">TotalWarnings</span> { <span style="color: #569cd6;">get</span>; <span style="color: #569cd6;">set</span>; }<span style="color: #6a9955;">//Set for Events:JobProgress,</span></div><div> <span style="color: #569cd6;">public</span> <span style="color: #569cd6;">double</span> <span style="color: #9cdcfe;">TotalDurationInMs</span> { <span style="color: #569cd6;">get</span>; <span style="color: #569cd6;">set</span>; }<span style="color: #6a9955;">//Set for Events:JobProgress,</span></div><div> <span style="color: #569cd6;">public</span> <span style="color: #569cd6;">int</span> <span style="color: #9cdcfe;">CpuDurationInMs</span> { <span style="color: #569cd6;">get</span>; <span style="color: #569cd6;">set</span>; }<span style="color: #6a9955;">//Set for Events:JobProgress,</span></div><div> <span style="color: #569cd6;">public</span> <span style="color: #569cd6;">int</span> <span style="color: #9cdcfe;">SqlDurationInMs</span> { <span style="color: #569cd6;">get</span>; <span style="color: #569cd6;">set</span>; }<span style="color: #6a9955;">//Set for Events:JobProgress,</span></div><div> <span style="color: #569cd6;">public</span> <span style="color: #569cd6;">int</span> <span style="color: #9cdcfe;">SqlQueryCount</span> { <span style="color: #569cd6;">get</span>; <span style="color: #569cd6;">set</span>; }<span style="color: #6a9955;">//Set for Events:JobProgress,</span></div><div> <span style="color: #569cd6;">public</span> <span style="color: #569cd6;">int</span> <span style="color: #9cdcfe;">WaitTimeOnSqlThrottlingMilliseconds</span> { <span style="color: #569cd6;">get</span>; <span style="color: #569cd6;">set</span>; }</div><div> <span style="color: #569cd6;">public</span> <span style="color: #569cd6;">string</span> <span style="color: #9cdcfe;">FileName</span> { <span style="color: #569cd6;">get</span>; <span style="color: #569cd6;">set</span>; }</div><div> <span style="color: #569cd6;">public</span> <span style="color: #569cd6;">bool</span> <span style="color: #9cdcfe;">IsShallowCopy</span> { <span style="color: #569cd6;">get</span>; <span style="color: #569cd6;">set</span>; }<span style="color: #6a9955;">//Set for Events:JobProgress,</span></div><div> <span style="color: #569cd6;">public</span> <span style="color: #4ec9b0;">Guid</span> <span style="color: #9cdcfe;">CorrelationId</span> { <span style="color: #569cd6;">get</span>; <span style="color: #569cd6;">set</span>; }</div><div> <span style="color: #569cd6;">public</span> <span style="color: #569cd6;">string</span> <span style="color: #9cdcfe;">SourceObjectFullUrl</span> { <span style="color: #569cd6;">get</span>; <span style="color: #569cd6;">set</span>; }</div><div> <span style="color: #569cd6;">public</span> <span style="color: #569cd6;">string</span> <span style="color: #9cdcfe;">TargetServerUrl</span> { <span style="color: #569cd6;">get</span>; <span style="color: #569cd6;">set</span>; }</div><div> <span style="color: #569cd6;">public</span> <span style="color: #569cd6;">string</span> <span style="color: #9cdcfe;">TargetListId</span> { <span style="color: #569cd6;">get</span>; <span style="color: #569cd6;">set</span>; }</div><div> <span style="color: #569cd6;">public</span> <span style="color: #569cd6;">string</span> <span style="color: #9cdcfe;">TargetObjectSiteRelativeUrl</span> { <span style="color: #569cd6;">get</span>; <span style="color: #569cd6;">set</span>; }</div><div> <span style="color: #569cd6;">public</span> <span style="color: #569cd6;">string</span> <span style="color: #9cdcfe;">TargetObjectUniqueId</span> { <span style="color: #569cd6;">get</span>; <span style="color: #569cd6;">set</span>; }</div><div> <span style="color: #569cd6;">public</span> <span style="color: #569cd6;">string</span> <span style="color: #9cdcfe;">TargetSiteName</span> { <span style="color: #569cd6;">get</span>; <span style="color: #569cd6;">set</span>; }</div><div> <span style="color: #6a9955;">// For JobStart</span></div><div> <span style="color: #569cd6;">public</span> <span style="color: #4ec9b0;">Guid</span> <span style="color: #9cdcfe;">FarmId</span> { <span style="color: #569cd6;">get</span>; <span style="color: #569cd6;">set</span>; }</div><div> <span style="color: #569cd6;">public</span> <span style="color: #4ec9b0;">Guid</span> <span style="color: #9cdcfe;">SubscriptionId</span> { <span style="color: #569cd6;">get</span>; <span style="color: #569cd6;">set</span>; }</div><div> <span style="color: #6a9955;">// For JobWarning</span></div><div> <span style="color: #569cd6;">public</span> <span style="color: #569cd6;">string</span> <span style="color: #9cdcfe;">ObjectType</span> { <span style="color: #569cd6;">get</span>; <span style="color: #569cd6;">set</span>; }</div><div> <span style="color: #569cd6;">public</span> <span style="color: #4ec9b0;">Guid</span> <span style="color: #9cdcfe;">Id</span> { <span style="color: #569cd6;">get</span>; <span style="color: #569cd6;">set</span>; }</div><div> <span style="color: #569cd6;">public</span> <span style="color: #569cd6;">int</span> <span style="color: #9cdcfe;">SourceListItemIntId</span> { <span style="color: #569cd6;">get</span>; <span style="color: #569cd6;">set</span>; }</div><div> <span style="color: #569cd6;">public</span> <span style="color: #569cd6;">int</span> <span style="color: #9cdcfe;">TargetListItemIntId</span> { <span style="color: #569cd6;">get</span>; <span style="color: #569cd6;">set</span>; }</div><div> <span style="color: #569cd6;">public</span> <span style="color: #569cd6;">string</span> <span style="color: #9cdcfe;">Message</span> { <span style="color: #569cd6;">get</span>; <span style="color: #569cd6;">set</span>; }</div><div> <span style="color: #569cd6;">public</span> <span style="color: #569cd6;">string</span> <span style="color: #9cdcfe;">ManifestFileName</span> { <span style="color: #569cd6;">get</span>; <span style="color: #569cd6;">set</span>; }<span style="color: #6a9955;">//Set for Events:FinishManifestFileUpload</span></div><div> }</div></div><p>Hope this is somewhat explanatory with good prior information about SharePoint CreateCopyJobs API. Planning to wrap all these into a working sample in GitHub. Hope that can be done soon.</p><p>Comments are welcome. Use this API at your risk as it lacks documentation and samples.</p><h2 style="text-align: left;">References</h2><div><a href="https://learn.microsoft.com/en-us/sharepoint/dev/apis/export-amr-api">https://learn.microsoft.com/en-us/sharepoint/dev/apis/export-amr-api</a> - About decryption</div>Joymonhttp://www.blogger.com/profile/12014569049635491702noreply@blogger.com0tag:blogger.com,1999:blog-3053230616754516408.post-33420784569620233402023-09-13T20:30:00.003-04:002023-09-13T20:30:00.152-04:00.Net Framework v/s .Net - One more performance comparisonSome of us might have heard why I should upgrade to .Net from legacy .Net Framework as it works and I want to run the application only on Windows servers. We as developers reply to them that .Net is modern and getting a lot of new features etc. Along with we emphasize that .Net is faster than .Net Framework. Should we use Linux to get that performance boost or will it be faster in Windows? There are a lot of comparisons<span style="background-color: #f8f9fa; color: #202122; font-family: sans-serif; font-size: 14px;">¹</span> available on the net regarding the same. Here is one more comparison.<div><h2 style="text-align: left;">Application use case</h2><div>This post is not to create a log of use cases and compare performance between the 2 frameworks. Rather this is comparing the encryption performance.</div><div>The application encrypts and decrypts data using X509Certificate. The use case tested may not be applicable in many projects but this is a solution to send data securely between the producers and one consumer. All the producers have the public key to encrypt and the consumer has the private key to decrypt.</div><div><br /></div><div>Source code: <a href="https://github.com/dotnet-demos/Org.Security.Cryptography.X509Extensions">https://github.com/dotnet-demos/Org.Security.Cryptography.X509Extensions</a></div><div><br /></div><div>This has projects that consume the shared code with minimal conditional compilations. Those are console apps and are ready to run with the required frameworks installed.</div><div><br /></div><div>Please create an issue or give a PR if anyone finds problems in the test code.</div><h2 style="text-align: left;">Test environment</h2><div>Model - HP EliteBook 840 G3</div><div>OS - Win 10 21H2 (OS Buid 19044.2194)</div><div>CPU - Intel i7-6600U @ 2.60Hz</div><div>RAM - 16GB</div><div>.Net Framework Version - 4.8(Runtime v4.0.30319)</div><div>.Net Version - .Net 6 (6.0.16)</div><div><h2 style="text-align: left;">Results</h2><div>The .Net performs twice the speed of the .Net Framework. The picture is worth 1,000 words</div><div class="separator" style="clear: both; text-align: center;"><a href="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/09/13/01-8kdata.png?raw=true" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="481" data-original-width="800" height="385" src="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/09/13/01-8kdata.png?raw=true" width="640" /></a></div><br /><div>The results were taken from an average of 3 attempts.</div><h2 style="text-align: left;">References</h2></div><div><span style="background-color: #f8f9fa; color: #202122; font-family: sans-serif; font-size: 14px;">¹</span><a href="https://softwarehut.com/blog/tech/net-core-vs-net-framework-testing-performance">https://softwarehut.com/blog/tech/net-core-vs-net-framework-testing-performance</a></div></div><div><br /></div>Joymonhttp://www.blogger.com/profile/12014569049635491702noreply@blogger.com0tag:blogger.com,1999:blog-3053230616754516408.post-38457010463936752812023-08-22T18:24:00.004-04:002023-08-22T18:24:56.433-04:00PowerShell to write XML with attributes on new line<h2 style="text-align: left;">Context</h2><div>Many times after patch deployment of the application into the server there will be questions on what changed? The source code is obviously changed and the binaries can't be compared to find out what changed.</div><div><br /></div><div>But the textual configuration files can easily be compared. Take a backup of existing files. Do the patching and compare what changed.</div><div><br /></div><div>This post is assuming the deployment tools are using PowerShell to change the config files.</div><h2 style="text-align: left;">Problem</h2><div>Look at the image below.<div class="separator" style="clear: both; text-align: center;"><a href="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/08/22/01-diff-format.png?raw=true" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="194" data-original-width="800" height="155" src="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/08/22/01-diff-format.png?raw=true" width="640" /></a></div><br /></div><div>It shows there are a lot of changes done to the file. But really there is only one change. <i>dbserver3->dbserver2</i></div><div><i><br /></i></div><div>As we can easily understand the comparison considered the files as text, not as XML, and found a lot of differences.</div><h2 style="text-align: left;">Approaches</h2><div>If the number of files is less this is fine but that is not the case in enterprise. There are multiple approaches to this problem.</div><h3 style="text-align: left;">A tool that understands XML and compares</h3><div>The above comparison tool is not recognizing the XML format instead it compared textually. Find another tool that understands the XML format and compare accordingly.</div><div><br /></div><div>To date, we didn't find any such tool. Please comment if there are tools that are free and export the comparison report.</div><h3 style="text-align: left;">Always have attributes on a new line</h3><div>In this approach make sure all the tools and programs that modify the XML write attributes on a new line.</div><div>Since PowerShell is running on top of .Net it is easy to get anything working in PowerShell that works in .Net. Google easily lands us in <a href="https://stackoverflow.com/questions/11266441/opening-saving-xml-while-preserving-newline-between-nodes-attributes" rel="nofollow" target="_blank">pages that </a>say use the XMLWriterSettings.NewLineOnAttributes to get the job done. Unfortunately, it is not working out of the box.</div><p>But when we combine the <i>.Indent</i> property with <i>.NewLineOnAttributes</i>, it works fine.</p><div style="background-color: #1f1f1f; color: #cccccc; font-family: Consolas, "Courier New", monospace; font-size: 14px; line-height: 19px; white-space: pre;"><div>[<span style="color: #569cd6;">XML</span>]<span style="color: #9cdcfe;">$doc</span> <span style="color: #d4d4d4;">=</span> <span style="color: #dcdcaa;">get-content</span> <span style="color: #ce9178;">".\input.xml"</span> </div><div><span style="color: #9cdcfe;">$xwSettings</span> <span style="color: #d4d4d4;">=</span> <span style="color: #dcdcaa;">new-object</span> System.Xml.XmlWriterSettings</div><div><span style="color: #9cdcfe;">$xwSettings</span><span style="color: #dcdcaa;">.Indent</span> <span style="color: #d4d4d4;">=</span> <span style="color: #569cd6;">$true</span></div><div><span style="color: #9cdcfe;">$xwSettings</span><span style="color: #dcdcaa;">.NewLineOnAttributes</span> <span style="color: #d4d4d4;">=</span> <span style="color: #569cd6;">$true</span></div><div><span style="color: #9cdcfe;">$xmlWriter</span> <span style="color: #d4d4d4;">=</span> [<span style="color: #569cd6;">Xml.XmlWriter</span>]::Create(<span style="color: #ce9178;">"output.xml"</span><span style="color: #d4d4d4;">,</span> <span style="color: #9cdcfe;">$xwSettings</span>)</div><div><span style="color: #9cdcfe;">$doc</span><span style="color: #dcdcaa;">.Save</span>(<span style="color: #9cdcfe;">$xmlWriter</span>)</div><div><span style="color: #9cdcfe;">$xmlWriter</span><span style="color: #dcdcaa;">.Dispose</span>()</div></div><div><br /></div><div>The above code works. Now the question is, is this documented? Oh yes in the <a href="https://learn.microsoft.com/en-us/dotnet/api/system.xml.xmlwritersettings.newlineonattributes?view=netframework-4.8#remarks" rel="nofollow" target="_blank">remarks section.</a><div class="separator" style="clear: both; text-align: center;"><a href="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/08/22/03-docs-remarks.png?raw=true" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="189" data-original-width="786" height="189" src="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/08/22/03-docs-remarks.png?raw=true" width="786" /></a></div></div><div>Of course, the default value of the <a href="https://learn.microsoft.com/en-us/dotnet/api/system.xml.xmlwritersettings.indent?view=netframework-4.8#property-value" rel="nofollow" target="_blank"><i>Indent</i> property is false</a>. The comparison looks as below after using the above code.</div><div><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/08/22/02-format.png?raw=true" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="199" data-original-width="800" height="159" src="https://github.com/blogfiles/blogfiles.github.io/blob/master/joymonscode/2023/08/22/02-format.png?raw=true" width="640" /></a></div><div><br /></div>Happy formatting.Joymonhttp://www.blogger.com/profile/12014569049635491702noreply@blogger.com0tag:blogger.com,1999:blog-3053230616754516408.post-85609269521162610832023-08-15T18:04:00.009-04:002023-09-13T15:03:40.333-04:00Azure @ Enterprise - Configuring PnP.Framework SDK logs to trace file<h2 style="text-align: left;">Context</h2><p>Environment - Microsoft Azure Virtual Machines</p><p>Technology - .Net Framework 4.8, SharePoint Online</p><p>Functionality - Interacting with SharePoint Online via Graph SDK and PnP.Framework, Mainly uploading a large number of files.</p><p>Requirement - Trying to collect the logs emitted by PnP.Framework especially the HTTP calls.</p><h2 style="text-align: left;">Solution</h2><div>The PnP.Framework emits the traces by default using the standard .Net Trace class. Note it is not the ILogger that is now the standard after .Net Framework. Refer<a href="https://github.com/pnp/pnpframework/blob/dev/src/lib/PnP.Framework/Diagnostics/Log.cs#L35" rel="nofollow" target="_blank"> to the source code</a> to ensure they change in the future. There is an <span style="background-color: red;">ILogger class involved but that is not the same .Net Core uses</span>. </div><div><ul style="text-align: left;"><li>.Net Core programming model - <i>Microsoft.Extensions.Logging.ILogger & Microsoft.Extensions.Logging.ILogger<TCategoryName></i></li><li>PnP.Framework - <i>PnP.Framework.Diagnostics.ILogger</i></li></ul></div><div>The default implementation of PnP.Framework is using TraceLogger class which is implemented by the same SDK. It uses the Trace class as mentioned above. So if we don't configure anything else, we can collect the logs to a file using the below configuration changes in app.config or web.config.<hr /></div><div><i><system.diagnostics></i></div><div><i> <trace autoflush="true" indentsize="1"></i></div><div><i> <listeners></i></div><div><i> <add name="Listener1" type="System.Diagnostics.TextWriterTraceListener" initializeData="path\to\file.ext" /></i></div><div><i> <remove name="Default" /></i></div><div><i> </listeners></i></div><div><i> </trace></i></div><div><i>... </i></div><div><i></system.diagnostics></i></div><span><hr />Here ends the basic solution to the problem. Anytime PnP.Framework SDK logs it will be saved to the file mentioned in the <listeners> section.</span><div><h3 style="text-align: left;"><span>Log Level</span></h3><div><span>By default the PnP.Framework logs at 'debug' which is strange.</span></div><script src="https://emgithub.com/embed-v2.js?target=https%3A%2F%2Fgithub.com%2Fpnp%2Fpnpframework%2Fblob%2Fdev%2Fsrc%2Flib%2FPnP.Framework%2FDiagnostics%2FLog.cs%23L82-90&style=default&type=code&showBorder=on&showLineNumbers=on&showFileMeta=on&showFullPath=on&showCopy=on"></script><div><span>But that can be changed with some adjustments. Config changes goes below<hr /></span></div><div><span><div><i><configSections></i></div><div><i> <sectionGroup name="pnp"></i></div><div><i> <section name="tracing" type="PnP.Framework.Diagnostics.LogConfigurationTracingSection,PnP.Framework"/></i></div><div><i> </sectionGroup></i></div><div><i> </configSections></i></div><div><i> <pnp></i></div><div><i> <tracing <span style="background-color: #fcff01;">logLevel="Debug"</span>></i></div><div><i> </tracing></i></div><div><i> </pnp></i><span><a name='more'></a>Basically this is adding a new section group and section that is read by the PnP.Framework. The logLevel attribute can be adjusted to control the logging level.</span></div></span></div><h2 style="text-align: left;">Enhancements</h2><div>Route the Traces to Azure Application Insights or any other telemetry collection system is recommended for production. There are already many ways to do the same as Trace was a standard way to log in class .Net Framework.</div><div><br /></div><div>If we need to customize the logging destination, we can even implement our own the <i>PnP.Framework.Diagnostics.ILogger</i> passes it via configuration. It is as follows<hr /></div><div><div><pnp></div><div> <tracing logLevel="Debug"></div><div> <logger type="MyAssembly.MyPnPFwkLogger,MyAssembly"></logger></div><div> </tracing></div><div> </pnp></div></div><hr /><div>Obviously, we need to code MyPnPFwkLogger class by implementing the ILogger interface. A sample implementation is below.<hr /></div><p> <i>class MyPnPFwkLogger : ILogger<br /> {<br /> void ILogger.Debug(LogEntry entry)<br /> {<br /> Console.WriteLine(entry);<br /> }<br /> void ILogger.Error(LogEntry entry)<br /> {<br /> Console.WriteLine(entry);<br /> }<br /> void ILogger.Info(LogEntry entry)<br /> {<br /> Console.WriteLine(entry);<br /> }<br /> void ILogger.Warning(LogEntry entry)<br /> {<br /> Console.WriteLine(entry);<br /> }<br /> }</i></p><p>This looks obvious, but it is very useful if we are seriously interacting with SharePoint Online via APIs. Many times the request-id is not present in other logs. Without request-id Microsoft support cannot locate the logs and solve problems.</p><p>Happy coding...</p></div>Joymonhttp://www.blogger.com/profile/12014569049635491702noreply@blogger.com0tag:blogger.com,1999:blog-3053230616754516408.post-48416112584630103222023-08-01T21:18:00.001-04:002023-08-01T21:18:13.116-04:00Raspberry Pi - Why my NAS (via SMB) is slow<p></p><h2>Problem</h2><div>As seen below, when I want to move files to and from my <a href="https://joymonscode.blogspot.com/search/label/NAS" rel="nofollow" target="_blank">NAS, I DIYed</a>. It was a little better before I moved to the new house.</div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEik1qLNvpkP0XUFzQpyTvbyfy6yIhLHNMauTLbIa9BQGVvt92lbJ1oW-Ya3tug9iFGrNe-H6aP1sQZRjsLLtGNG9HQs9sDeA3rx-E6SCGk24T53ER0KJO0LeH263Boz23EfUmoVlC1csprJL8_UR-y6DEWoWWjkDTNmsiQYnObsXSlNa_Io_ll90Wyict4" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="246" data-original-width="387" height="203" src="https://blogger.googleusercontent.com/img/a/AVvXsEik1qLNvpkP0XUFzQpyTvbyfy6yIhLHNMauTLbIa9BQGVvt92lbJ1oW-Ya3tug9iFGrNe-H6aP1sQZRjsLLtGNG9HQs9sDeA3rx-E6SCGk24T53ER0KJO0LeH263Boz23EfUmoVlC1csprJL8_UR-y6DEWoWWjkDTNmsiQYnObsXSlNa_Io_ll90Wyict4" width="320" /></a></div><h2 style="text-align: left;">Specs</h2><ul style="text-align: left;"><li>Raspberry Pi 4 4GB is hosting the NAS</li><li>Both RasPi and client Windows machines are in WiFi 5 (In the old 1BHK apartment all were nearby and used LAN cable)</li><li>RasPi is hosting another service such as Plex.</li><li>Seagate Backup Plus 8TB HDD (not SSD) connected to usb3</li><li>HDD uses NTFS file system than Linux native etx4</li><li>Uses SMB3 protocol</li></ul><div>There are many guesses or options people are talking about on the internet, from converting NTFS to ext4 to buying a dedicated NAS device, etc...Let us go the <a href="https://www.youtube.com/watch?app=desktop&v=jcf0_nbcZyM" rel="nofollow" target="_blank">scientific debugging</a> route.</div><h2 style="text-align: left;">Hypotheses</h2><h3 style="text-align: left;">Slow HDD access</h3><div style="text-align: left;">There is a bottleneck in the connection between HDD and the RasPi.</div><h3 style="text-align: left;">Testing</h3><div>There is a utility called hdparam in Linux to test the connectivity. Below goes the result.</div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEi-fQ8ZMtcgnOeU6aW10XeaJqduymV4IEv_uYRG7KoWfEZiEso32a6GCubwFpHhSfOJOfVIzSFKOJ39Mu6l7Uja87ARpH-bxp1xd9yCv5NDAhrws2pe3yNlg-vxc8Rx3OpvP726dloZdeJ8QNS8tePF_ObdWxiSnzWQIH07gSjQ_3SV3zmWLGFsPgJpgko" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="99" data-original-width="1050" height="60" src="https://blogger.googleusercontent.com/img/a/AVvXsEi-fQ8ZMtcgnOeU6aW10XeaJqduymV4IEv_uYRG7KoWfEZiEso32a6GCubwFpHhSfOJOfVIzSFKOJ39Mu6l7Uja87ARpH-bxp1xd9yCv5NDAhrws2pe3yNlg-vxc8Rx3OpvP726dloZdeJ8QNS8tePF_ObdWxiSnzWQIH07gSjQ_3SV3zmWLGFsPgJpgko=w640-h60" width="640" /></a></div><h3 style="clear: both; text-align: left;">Status</h3><div>Rejected as it is showing better numbers.</div><h2 style="clear: both; text-align: left;">Slow network connection</h2><div>Due to the WiFi 5, it is slow in communicating between machines</div><h3 style="text-align: left;">Testing</h3><div>There is a utility called iperf3 to test the network speed.</div><div><br /></div>iperf3 to local network<br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEhmlVuOTxnhVepN1-yM7d5oaKZaa0Yy-sM3RZbWgcXF5f0H7ZXxcYhCDzRTi0qJs3qhqjw2Vy_U3e6Janhcz9934pAv3jj2S6kHyo6Aw2FkrcpZESZEP-eNeBnl-24RZo96jhpBwmA6v-0uAsJJGSNnU7GOfBBpmYvFRnQwtLa4ll3LYbyhQsnSIe17BoE" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="331" data-original-width="614" height="346" src="https://blogger.googleusercontent.com/img/a/AVvXsEhmlVuOTxnhVepN1-yM7d5oaKZaa0Yy-sM3RZbWgcXF5f0H7ZXxcYhCDzRTi0qJs3qhqjw2Vy_U3e6Janhcz9934pAv3jj2S6kHyo6Aw2FkrcpZESZEP-eNeBnl-24RZo96jhpBwmA6v-0uAsJJGSNnU7GOfBBpmYvFRnQwtLa4ll3LYbyhQsnSIe17BoE=w640-h346" width="640" /></a></div><div class="separator" style="clear: both;">iperf3 test to a public server</div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEhXDUkOwdYfk4nOyB1vuZdFNOsn7I4UJUKXZ59RTLubH4IcVZIaqBW_LQwcProlkE4JyBUDsqqtjA3ANAOXOYLAV-MnOOZyYDrHHEDWcUQjXcRlm61faJbT8TSZ9JG0u7AAMVBGsOHuInFZK37zTByJcnSWmWkDZd6vvqX-IT_qOavi1dnLugubv5WAypY" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="329" data-original-width="654" height="322" src="https://blogger.googleusercontent.com/img/a/AVvXsEhXDUkOwdYfk4nOyB1vuZdFNOsn7I4UJUKXZ59RTLubH4IcVZIaqBW_LQwcProlkE4JyBUDsqqtjA3ANAOXOYLAV-MnOOZyYDrHHEDWcUQjXcRlm61faJbT8TSZ9JG0u7AAMVBGsOHuInFZK37zTByJcnSWmWkDZd6vvqX-IT_qOavi1dnLugubv5WAypY=w640-h322" width="640" /></a></div><div><br /></div><h3 style="text-align: left;">Result</h3><div>The hypothesis is true that connectivity is the bottleneck. Either upgrade to use WiFi 6 which RasPi 4 doesn't support out of the box as well as most of my machines.</div><h2 style="text-align: left;">Next steps</h2><div>Have some way to connect the RasPi and possible machines via ethernet cable. The wiring may be required.<div class="separator" style="clear: both; text-align: center;"><br /><p></p></div></div>Joymonhttp://www.blogger.com/profile/12014569049635491702noreply@blogger.com0tag:blogger.com,1999:blog-3053230616754516408.post-66811779743672333842023-06-27T22:00:00.001-04:002023-06-27T22:00:00.161-04:00Azure @ Enterprise - Upload certificate to App registration using PowerShell<p style="text-align: center;"><span style="font-size: large;">Warning</span></p><blockquote>All characters and events in this post are fictional. Any resemblance or similarity to any actual events entities or persons is entirely coincidental</blockquote><p></p><p><i>Developer: Hi there, as you know we are excited to release the next version that uses Azure AD-based authentication. All the 100 development, testing till prod deployment instances require their own Azure app registrations. Also, require certificates added for authentication to outside services. Certificates can be shared across lower environments.</i></p><p>Install Manager: Oh. It is not possible to have one app registration per application instance. It is not manageable.</p><p><i>Developer: What? Are you not paid for managing instances? Hope you are not doing charity.</i></p><p>Install Manager: Yeah I know you are funny. We cannot manage that many app registrations and certificates.</p><p><i>Developer: What do you mean by managing? </i></p><p>Install Manager: Oh boy, higher environments are not like your toy dev instances. It is serious business. Do you know the certificates are not forever? On that day they expire, we need to upload a new certificate to all your dev and test app registrations.</p><p><i>Developer: Hey, we are in 2023. Don't you guys automate?</i></p><p>Install Manager: We don't do automation here.</p><p><i>Developer: It is not huge code. Just use the simple PowerShell commands that any system admin knows.</i></p><p>Install Manager: Enough boy. We don't have skilled people here. If you want, you code it and give it to us.</p><p><i>Developer: Oh ok..<a href="https://github.com/ms-azure-demos/app-registration-upload-certificate-powershell" rel="nofollow" target="_blank">here you go</a>.</i></p>Joymonhttp://www.blogger.com/profile/12014569049635491702noreply@blogger.com0tag:blogger.com,1999:blog-3053230616754516408.post-27473320747157529612023-06-20T10:30:00.007-04:002023-06-20T10:30:00.145-04:00Mind map of SharePoint Online APIs and .Net SDK model<p>This relates somewhat to a post in this blog about the <a href="https://joymonscode.blogspot.com/2022/05/dilemma-of-choosing-net-sdk-to-interact.html" rel="nofollow" target="_blank">dilemma of choosing Net SDK to interact with SharePoint</a>". Since there are multiple .Net SDKs the selection of one is tricky. Now think about multiple APIs to interact with. For simplicity, we are now concerned about only SharePoint Online by avoiding SharePoint services on-premises.</p><p>This post tries to visually represent the APIs and corresponding .Net SDK to interact with SharePoint Online.</p><p><br /><a href="https://www.plantuml.com/plantuml/proxy?fmt=svg&cache=no&src=https://raw.githubusercontent.com/mind-maps/software/master/programming/microsoft-365/sharepoint/api-dotnet-sdk.puml" rel="nofollow" style="margin-left: 1em; margin-right: 1em;" target="_blank"><img border="0" data-original-height="519" data-original-width="800" height="170" src="https://www.plantuml.com/plantuml/proxy?fmt=png&cache=no&src=https://raw.githubusercontent.com/mind-maps/software/master/programming/microsoft-365/sharepoint/api-dotnet-sdk.puml" width="640" /></a></p><p><span style="font-size: x-small;">Generated using PlantUML. Click on the image to get svc where the links are navigatable.</span></p><p>Good luck interacting with SharePoint.</p>Joymonhttp://www.blogger.com/profile/12014569049635491702noreply@blogger.com0tag:blogger.com,1999:blog-3053230616754516408.post-74546616370453044852023-06-13T10:50:00.001-04:002023-06-13T10:50:00.152-04:00Understanding the throttling in SharePoint Online<p style="text-align: justify;">As mentioned many times in this blog, it is very easy to use SharePoint for document management and to maintain a small list without writing a custom application. But it is a nightmare when it comes to customizing SharePoint and interacting with it from other applications from APIs. One of the challenges is the throttling behavior. </p><p>We should agree that it is a SaaS application and Microsoft cannot fulfill all our API requests. They should throttle for the below reasons</p><p></p><ul style="text-align: left;"><li>In order to manage noisy neighbors give equal consideration to all the tenant</li><li>Ensure the availability of the entire system</li></ul><div>The problem is the throttling details are not clear from the documentation. None available if we ended up in a situation where we need to use user-based flow. Forget about using ROPC flow from a daemon application. </div><div>Below is a flow chart that is trying to make sense of when our application will get throttled by Microsoft.<div class="separator" style="clear: both; text-align: center;"><a href="https://www.plantuml.com/plantuml/proxy?fmt=svg&cache=no&src=https://raw.githubusercontent.com/mind-maps/software/master/programming/microsoft-365/sharepoint/throttling-flow.puml" imageanchor="1" rel="nofollow" style="margin-left: 1em; margin-right: 1em;" target="_blank"><img border="0" data-original-height="519" data-original-width="800" height="415" src="https://www.plantuml.com/plantuml/proxy?fmt=png&cache=no&src=https://raw.githubusercontent.com/mind-maps/software/master/programming/microsoft-365/sharepoint/throttling-flow.puml" width="640" /></a></div>Click on it to open as SVG to navigate to the links.</div><p></p>Joymonhttp://www.blogger.com/profile/12014569049635491702noreply@blogger.com0